Testing best practices opinion needed on testing assets

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Testing best practices opinion needed on testing assets

Ralph Schindler-2
Hey all!

As you might have seen in recent tweets and mailing list posts, we have
been in the process of converting the current codebase to namespaces,
this includes library files and test files.

In converting the test files, I've run across an issue I'd like to get
feedback on.  "What to do with test assets?"

What to I mean by that?

Well, currently we've used the best practice of adding needed testing
assets into a _files directory.  While this works fine, one of the goals
moving forward into php 5.3 is to remove the need for require_once
calls.  In the spirit of that objective, I propose that when creating
assets specifically for testing, we use a sub namespace called TestAsset
to contain these assets.  For example:

If a component needs Mock classes, or Dummy classes to test against or
consume during testing, they would be located in a TestAsset namespace.

For example, if the component is Zend\Tag\Cloud, the testing namespace
for this component would be ZendTest\Tag\Cloud, and inside that
namespace would exist ZendTest\Tag\Cloud\TestAsset.

To get a sense of what this would look like, please see:

http://git.mwop.net/?a=viewblob&p=zf&h=c61faa7a98af7874c8676aa30fcd7f459a18d711&hb=52e34018d73bfa2a08a63af2caab9d78f71231c2&f=tests/Zend/Tag/Cloud/TestAsset/CloudDummy.php

I propose this would be a good best practice to move forward with since
it will allow us to create code that is autoloadable in the test
directory, and follows a clean coding convention similar to what we have
in other respects (library code).

There are no PHPUnit conventions to my knowledge that address assets...

What do you think?

-Ralph
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

Giorgio Sironi
This is indeed a good solution. The only alternative would be
incorporating classes into the sourcefile of the test case, either
with the definition of other classes or via self-shunting of the test
case class. Since in PHP there is no support for private/anonymous
classes, I guess a subnamespace is ok.

--
Giorgio Sironi
Piccolo Principe & Web Engineer
http://giorgiosironi.blogspot.com
http://twitter.com/giorgiosironi
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

Ralph Schindler-2
This actually brings up another point in line with this suggested best
practice.

I think we should stick with the 1 class per file rule in tests.  This
makes it easy to identify the containing file of the target class.  I've
come across lots of test cases that have classes tacked onto the bottom
of the test case file, and IMO, its not as "clean" of a solution and I
personally would like.

Thoughts there?

-ralph

Giorgio Sironi wrote:
> This is indeed a good solution. The only alternative would be
> incorporating classes into the sourcefile of the test case, either
> with the definition of other classes or via self-shunting of the test
> case class. Since in PHP there is no support for private/anonymous
> classes, I guess a subnamespace is ok.
>
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

hobodave
In reply to this post by Ralph Schindler-2
This seems sensible to me. I also agree with the single class per file rule. Our application code follows this convention; why should tests be different?

--
David Abdemoulaie
http://hobodave.com/
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

A.J. Brown-3
That sounds like a great solution.

Cheers!

On Tue, Apr 13, 2010 at 4:52 PM, David Abdemoulaie <[hidden email]> wrote:
> This seems sensible to me. I also agree with the single class per file rule. Our application code follows this convention; why should tests be different?
>
> --
> David Abdemoulaie
> http://hobodave.com/



--
A.J. Brown
Software Engineer, ZCE
blog : http://ajbrown.org
talk  : (937) 540-0099
chat : IntypicaAJ
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

Dolf Schimmel
I'm not sure we should use the ZendTest namespace as primary ns. What
would be (good(!)) reasons to not put it in Zend\TestSuite\ (which is
what Matthew mentioned long, long ago). Imho it would make sense to
put the testsuite in a subpackage since the testsuite is part of ZF.
Alternatively, we could do it like Doctrine does: put the library in
Zend\Library\ and the testsuite in Zend\TestSuite. Of course I
understand this would probably be a bit unpractical (and late), but it
would be the neatest way of achieving this all.

My $0.02 (or whatever value you like, it's a matter of speech anyways).

Dolf
Freeaqingme


On Wed, Apr 14, 2010 at 2:10 AM, A.J. Brown <[hidden email]> wrote:

> That sounds like a great solution.
>
> Cheers!
>
> On Tue, Apr 13, 2010 at 4:52 PM, David Abdemoulaie <[hidden email]> wrote:
>> This seems sensible to me. I also agree with the single class per file rule. Our application code follows this convention; why should tests be different?
>>
>> --
>> David Abdemoulaie
>> http://hobodave.com/
>
>
>
> --
> A.J. Brown
> Software Engineer, ZCE
> blog : http://ajbrown.org
> talk  : (937) 540-0099
> chat : IntypicaAJ
>
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

till
In reply to this post by Ralph Schindler-2
On Tue, Apr 13, 2010 at 8:22 PM, Ralph Schindler
<[hidden email]> wrote:

> Hey all!
>
> As you might have seen in recent tweets and mailing list posts, we have been
> in the process of converting the current codebase to namespaces, this
> includes library files and test files.
>
> In converting the test files, I've run across an issue I'd like to get
> feedback on.  "What to do with test assets?"
>
> What to I mean by that?
>
> Well, currently we've used the best practice of adding needed testing assets
> into a _files directory.  While this works fine, one of the goals moving
> forward into php 5.3 is to remove the need for require_once calls.  In the
> spirit of that objective, I propose that when creating assets specifically
> for testing, we use a sub namespace called TestAsset to contain these
> assets.  For example:
>
> If a component needs Mock classes, or Dummy classes to test against or
> consume during testing, they would be located in a TestAsset namespace.
>
> For example, if the component is Zend\Tag\Cloud, the testing namespace for
> this component would be ZendTest\Tag\Cloud, and inside that namespace would
> exist ZendTest\Tag\Cloud\TestAsset.
>
> To get a sense of what this would look like, please see:
>
> http://git.mwop.net/?a=viewblob&p=zf&h=c61faa7a98af7874c8676aa30fcd7f459a18d711&hb=52e34018d73bfa2a08a63af2caab9d78f71231c2&f=tests/Zend/Tag/Cloud/TestAsset/CloudDummy.php
>
> I propose this would be a good best practice to move forward with since it
> will allow us to create code that is autoloadable in the test directory, and
> follows a clean coding convention similar to what we have in other respects
> (library code).
>
> There are no PHPUnit conventions to my knowledge that address assets...
>
> What do you think?
>
> -Ralph
>

On require_once:

I think stripping them from the framework is a good idea, but on the
test suite -- isn't it also overcomplicating things?
I mean, for me stripping require_once calls is primarily a performance
thing. I've been doing it in production code for over a year or so
now. But I've never done it inside my test suite. Part of this is --
it doesn't really matter how many milliseconds I gain here, especially
not at the expense of more sophisticated development and "best
practices".

I'd rather prefer slow straight-forward tests.

On the namespaces for assets:

I think a standardized namespace in Zend\Test\Component\Asset (or
similar) would be a great idea. One concern is though -- do I always
have class code as assets, or sometimes just files? For example, let's
assume I want to test gettext files -- do I need to wrap those into
class code, or could I still provide a dummy file to work with?

I also agree with Dolf (and Matthew) on Zend\TestSuite\ vs Zend\Test.

Till
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

Giorgio Sironi
On Wed, Apr 14, 2010 at 12:43 PM, till <[hidden email]> wrote:

> On require_once:
>
> I think stripping them from the framework is a good idea, but on the
> test suite -- isn't it also overcomplicating things?
> I mean, for me stripping require_once calls is primarily a performance
> thing. I've been doing it in production code for over a year or so
> now. But I've never done it inside my test suite. Part of this is --
> it doesn't really matter how many milliseconds I gain here, especially
> not at the expense of more sophisticated development and "best
> practices".

It's also a maintenance burden, more than a performance problem.
require_once() statements have to be synchronized with the code.

> I think a standardized namespace in Zend\Test\Component\Asset (or
> similar) would be a great idea. One concern is though -- do I always
> have class code as assets, or sometimes just files? For example, let's
> assume I want to test gettext files -- do I need to wrap those into
> class code, or could I still provide a dummy file to work with?

Autoloading works for class definitions, so I guess xml, ini, etc.
files can be put in the Asset folder but has to be loaded manually
(and usually not even with require_once() but by the production code
under test.)

> I also agree with Dolf (and Matthew) on Zend\TestSuite\ vs Zend\Test.

Haven't a parallel hierarchy been considered? Zend\Component\ClassTest
for testing Zend\Component\Class (of course all files in the test/
directory instead of lib/.)

--
Giorgio Sironi
Piccolo Principe & Web Engineer
http://giorgiosironi.blogspot.com
http://twitter.com/giorgiosironi
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

till
On Wed, Apr 14, 2010 at 1:04 PM, Giorgio Sironi
<[hidden email]> wrote:

> On Wed, Apr 14, 2010 at 12:43 PM, till <[hidden email]> wrote:
>> On require_once:
>>
>> I think stripping them from the framework is a good idea, but on the
>> test suite -- isn't it also overcomplicating things?
>> I mean, for me stripping require_once calls is primarily a performance
>> thing. I've been doing it in production code for over a year or so
>> now. But I've never done it inside my test suite. Part of this is --
>> it doesn't really matter how many milliseconds I gain here, especially
>> not at the expense of more sophisticated development and "best
>> practices".
>
> It's also a maintenance burden, more than a performance problem.
> require_once() statements have to be synchronized with the code.

I don't really agree with that. It's not like everyone plans to
reshuffle the structure every week.

>> I think a standardized namespace in Zend\Test\Component\Asset (or
>> similar) would be a great idea. One concern is though -- do I always
>> have class code as assets, or sometimes just files? For example, let's
>> assume I want to test gettext files -- do I need to wrap those into
>> class code, or could I still provide a dummy file to work with?
>
> Autoloading works for class definitions, so I guess xml, ini, etc.
> files can be put in the Asset folder but has to be loaded manually
> (and usually not even with require_once() but by the production code
> under test.)

I wasn't suggesting we use require_once for an .xml file. I was just
saying that namespace might not work for everything there is. ;-) So a
_files directory might still be necessary.

Till
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

padraicb
In reply to this post by Giorgio Sironi
>Haven't a parallel hierarchy been considered? Zend\Component\ClassTest
>for testing Zend\Component\Class (of course all files in the test/
>directory instead of lib/.)

Depends on what you mean by a parallel hierarchy. A basic hierarchy, as we have, is sufficient (which seems to be what you mean) but we shouldn't start specifying specific hierarchies outside of the basic component path. Testers should be free to aggregate and double up on test classes compared to the units under test.

Paddy
 
Pádraic Brady

http://blog.astrumfutura.com
http://www.survivethedeepend.com
OpenID Europe Foundation Irish Representative



From: Giorgio Sironi <[hidden email]>
To: [hidden email]
Sent: Wed, April 14, 2010 12:04:51 PM
Subject: Re: [zf-contributors] Testing best practices opinion needed on testing assets

On Wed, Apr 14, 2010 at 12:43 PM, till <[hidden email]> wrote:

> On require_once:
>
> I think stripping them from the framework is a good idea, but on the
> test suite -- isn't it also overcomplicating things?
> I mean, for me stripping require_once calls is primarily a performance
> thing. I've been doing it in production code for over a year or so
> now. But I've never done it inside my test suite. Part of this is --
> it doesn't really matter how many milliseconds I gain here, especially
> not at the expense of more sophisticated development and "best
> practices".

It's also a maintenance burden, more than a performance problem.
require_once() statements have to be synchronized with the code.

> I think a standardized namespace in Zend\Test\Component\Asset (or
> similar) would be a great idea. One concern is though -- do I always
> have class code as assets, or sometimes just files? For example, let's
> assume I want to test gettext files -- do I need to wrap those into
> class code, or could I still provide a dummy file to work with?

Autoloading works for class definitions, so I guess xml, ini, etc.
files can be put in the Asset folder but has to be loaded manually
(and usually not even with require_once() but by the production code
under test.)

> I also agree with Dolf (and Matthew) on Zend\TestSuite\ vs Zend\Test.

Haven't a parallel hierarchy been considered? Zend\Component\ClassTest
for testing Zend\Component\Class (of course all files in the test/
directory instead of lib/.)

--
Giorgio Sironi
Piccolo Principe & Web Engineer
http://giorgiosironi.blogspot.com
http://twitter.com/giorgiosironi
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

padraicb
In reply to this post by Ralph Schindler-2
I'm supportive of any guiding principles, but I think it needs it needs saying that we're talking about tests. Based on my own approach to testing, it's unlikely I'd spend time avoiding require_once calls and in-file classes (I barely even bother with comment blocks as is). Just to explain that a bit more, my own approach is a lot more "short-viewed" and unitised than the prevailing practice so I write the test, pass it, and then forget about it (faster the better)...completely until/unless it breaks. My largest test suites run to hundreds of tests (see Zend_Feed for more of this insanity ;)). Splitting out classes and watching how I include resources falls into the "nice to have" part of that whereas in our production classes they are essentially inherent in the design already. Although they are nice to have, they're not essential, and when you have hundreds of tests, not likely to ever appear.

If we're going to hold tests to the same standards as their tested units, it's going to detract from the loose and fast "get it done now" tests and increase the burden on testers. Given our existing problems with quality tests, and we have some serious problems there to address for ZF 2.0, I'm not sure we want to pile on too many requirements that make little functional difference.

If anything, I'd prefer we focus more on quality than cleanliness. Cleanliness is meaningless without quality IMO. I'd rate the importance of a proper test name quite a bit above whether or not the test used an in-file or namespaced class that took the tester extra time to separate.

It should be noted my opinion is just my opinion. I also believe code coverage is a complete waste of time, our tests too complicated and interrelated, and we should all convert to worshipping the Spaghetti Monster ;).

Paddy 
 
Pádraic Brady

http://blog.astrumfutura.com
http://www.survivethedeepend.com
OpenID Europe Foundation Irish Representative



From: Ralph Schindler <[hidden email]>
To: Giorgio Sironi <[hidden email]>
Cc: "Zend Framework Contributors ([hidden email])" <[hidden email]>
Sent: Tue, April 13, 2010 9:43:10 PM
Subject: Re: [zf-contributors] Testing best practices opinion needed on testing assets

This actually brings up another point in line with this suggested best practice.

I think we should stick with the 1 class per file rule in tests.  This makes it easy to identify the containing file of the target class.  I've come across lots of test cases that have classes tacked onto the bottom of the test case file, and IMO, its not as "clean" of a solution and I personally would like.

Thoughts there?

-ralph

Giorgio Sironi wrote:
> This is indeed a good solution. The only alternative would be
> incorporating classes into the sourcefile of the test case, either
> with the definition of other classes or via self-shunting of the test
> case class. Since in PHP there is no support for private/anonymous
> classes, I guess a subnamespace is ok.
>
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

weierophinney
Administrator
-- Pádraic Brady <[hidden email]> wrote
(on Wednesday, 14 April 2010, 05:13 AM -0700):

> I'm supportive of any guiding principles, but I think it needs it needs saying
> that we're talking about tests. Based on my own approach to testing, it's
> unlikely I'd spend time avoiding require_once calls and in-file classes (I
> barely even bother with comment blocks as is). Just to explain that a bit more,
> my own approach is a lot more "short-viewed" and unitised than the prevailing
> practice so I write the test, pass it, and then forget about it (faster the
> better)...completely until/unless it breaks. My largest test suites run to
> hundreds of tests (see Zend_Feed for more of this insanity ;)). Splitting out
> classes and watching how I include resources falls into the "nice to have" part
> of that whereas in our production classes they are essentially inherent in the
> design already. Although they are nice to have, they're not essential, and when
> you have hundreds of tests, not likely to ever appear.
>
> If we're going to hold tests to the same standards as their tested units, it's
> going to detract from the loose and fast "get it done now" tests and increase
> the burden on testers. Given our existing problems with quality tests, and we
> have some serious problems there to address for ZF 2.0, I'm not sure we want to
> pile on too many requirements that make little functional difference.
>
> If anything, I'd prefer we focus more on quality than cleanliness. Cleanliness
> is meaningless without quality IMO. I'd rate the importance of a proper test
> name quite a bit above whether or not the test used an in-file or namespaced
> class that took the tester extra time to separate.


I tend to agree with Paddy here.

When testing, I want to be agile: what is the behavior I'm testing, and
how do I express that in code succinctly? If I can simply embed another
class in the same file, or even another namespace, that's often faster
and easier than creating an additional file or files for doing so. If I
want to move a mock or stub class out into a separate file, it's easy
enough to do, and I can either do a require_once, "use" statement, or
simply instantiate the class (assuming it's in the same namespace or a
sub-namespace).

I think some loose guidelines may be in order, but I don't think they
should be hard directives; if they get in the way of efficiently
creating good tests, they should be ignored.


> It should be noted my opinion is just my opinion. I also believe code
> coverage is a complete waste of time, our tests too complicated and
> interrelated, and we should all convert to worshipping the Spaghetti
> Monster ;).
>
> Paddy
>  
> P draic Brady
>
> http://blog.astrumfutura.com
> http://www.survivethedeepend.com
> OpenID Europe Foundation Irish Representative
>
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> From: Ralph Schindler <[hidden email]>
> To: Giorgio Sironi <[hidden email]>
> Cc: "Zend Framework Contributors ([hidden email])"
> <[hidden email]>
> Sent: Tue, April 13, 2010 9:43:10 PM
> Subject: Re: [zf-contributors] Testing best practices opinion needed on testing
> assets
>
> This actually brings up another point in line with this suggested best
> practice.
>
> I think we should stick with the 1 class per file rule in tests.  This makes it
> easy to identify the containing file of the target class.  I've come across
> lots of test cases that have classes tacked onto the bottom of the test case
> file, and IMO, its not as "clean" of a solution and I personally would like.
>
> Thoughts there?
>
> -ralph
>
> Giorgio Sironi wrote:
> > This is indeed a good solution. The only alternative would be
> > incorporating classes into the sourcefile of the test case, either
> > with the definition of other classes or via self-shunting of the test
> > case class. Since in PHP there is no support for private/anonymous
> > classes, I guess a subnamespace is ok.
> >

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

Ralph Schindler-2

>> If anything, I'd prefer we focus more on quality than cleanliness. Cleanliness
>> is meaningless without quality IMO. I'd rate the importance of a proper test
>> name quite a bit above whether or not the test used an in-file or namespaced
>> class that took the tester extra time to separate.

I agree with you here.  The basis of the argument is not for quality
(which is important) but for maintenance.  If there is a "loose" rule of
where to expect to find assets, then *other* developers that want to
contribute will have an expectation of where to find things when digging
into the tests.

> When testing, I want to be agile: what is the behavior I'm testing, and
> how do I express that in code succinctly? If I can simply embed another
> class in the same file, or even another namespace, that's often faster

Sidenote: what might be succinct for one developer (using what they'd
consider to be obvious placements and/or practices) might not be as
apparent to another.

> I think some loose guidelines may be in order, but I don't think they
> should be hard directives; if they get in the way of efficiently
> creating good tests, they should be ignored.

I like the term loose guidelines. I'd say ideally if there are LOTS of
assets being created, they should generally go into a TestAsset namespace.

Ideally, I think putting them inside the same file as the test should be
a no-go though. In most developers heads already is the concept that
there is a 1-1 mapping of class to file.  If a developer ever wanted
track down a reference, with the 1-1 rule, it's clear where their first
look would be.  If it's not there, currently you have to go poking
through all of the components code to try and identify the proper source.

>> It should be noted my opinion is just my opinion. I also believe code
>> coverage is a complete waste of time, our tests too complicated and
>> interrelated, and we should all convert to worshipping the Spaghetti
>> Monster ;).

Interesting point.. my opinion here is based on the fact that ZF is
developed and maintained by a community of people, not just a few core
developers.  Code coverage is a hard and fast rule that forces people to
get to some level of conformance.  Conformance is good when it comes to
getting new developers up to speed.  This is also the same driving
factor that drives the opinion that keeping assets in a known place will
make it easier for new developers to just "jump in" and start contributing.

-ralph
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

Ralph Schindler-2
In reply to this post by till

> On require_once:
>
> I think stripping them from the framework is a good idea, but on the
> test suite -- isn't it also overcomplicating things?
> I mean, for me stripping require_once calls is primarily a performance
> thing. I've been doing it in production code for over a year or so
> now. But I've never done it inside my test suite. Part of this is --
> it doesn't really matter how many milliseconds I gain here, especially
> not at the expense of more sophisticated development and "best
> practices".
>
> I'd rather prefer slow straight-forward tests.

It's actually not about performance at all, it's about the ability to
find things where one might expect them to be.  This, above all else,
speaks towards maintenance and ability for new developers to understand
where things are expected to go, and where current things are expected
to be.

This is also supported by future infrastructure.. It is (will be) easy
to have everything loadable by SPLClassLoader.

http://wiki.php.net/rfc/splclassloader

> On the namespaces for assets:
>
> I think a standardized namespace in Zend\Test\Component\Asset (or
> similar) would be a great idea. One concern is though -- do I always
> have class code as assets, or sometimes just files? For example, let's
> assume I want to test gettext files -- do I need to wrap those into
> class code, or could I still provide a dummy file to work with?
>
> I also agree with Dolf (and Matthew) on Zend\TestSuite\ vs Zend\Test.
>
> Till
>

Non PHP assets will still need to be fopen()'d or the one-off .php files
that dont contain classes will need to be include()'d.

To reiterate, the main objective is to make the tests of Zend Framework
easier to understand for all of the contributors, new and old.  Is it
really that hard and/or create a new file per asset when you are
testing? It's what you would have done in your library code anyway, and
its only a few clicks here and there ;)

-ralph
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

Ralph Schindler-2
In reply to this post by Dolf Schimmel


Actually (the name ZendTest at the top level) I feel is important to
make the immediate distinction between requirements.  Anything in
ZendTest has a hard requirement on the PHPUnit Framework.  Also, if
looking at code in an API list or an IDE, if both the library classes
and test classes are listed, you will not have testing API's (public
members, and Test classes) listed in with the library code.

When I look at library/, I see files that contain classes that in each
exports a certain public API for consumption by someone who "wants to
use ZF".

Tests are something entirely different, and in most cases, typical
consumers of ZF do not want the tests on their system (for them it makes
no sense).  In few words, tests have no place in the library.  I think
that is the predominant reason why you'd see tests/ and library/ as
separate directories in a project.  Put yet another way, tests/* are
full of code that consume the library- just like a developer would
consume the library/ from their application.

> put the testsuite in a subpackage since the testsuite is part of ZF.

What is this subpackage you speak of? ;)

The testsuite is NOT part of the "ZF Library" though.  Just as
documentation is not in library/ and tutorials are not in the library/,
and you dont build your application inside the library/ (similar to the
download-and-go style distributions of frameworks out there, Cake i think?).

-ralph
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

padraicb
In reply to this post by Ralph Schindler-2
>> If anything, I'd prefer we focus more on quality than cleanliness. Cleanliness
>>> is meaningless without quality IMO. I'd rate the importance of a proper test
>>> name quite a bit above whether or not the test used an in-file or namespaced
>>> class that took the tester extra time to separate.
>
>I agree with you here.  The basis of the argument is not for quality (which is important) but for
>maintenance.  If there is a "loose" rule of where to expect to find assets, then *other*
>developers that want to contribute will have an expectation of where to find things when digging
>into the tests.

It's a difference of opinion, when I associate maintenance with unit tests it's going to drag concepts from quality of such tests. For example, many unit tests at present are highly aggregate generic tests (see Zend_Feed's original set). The characteristics of these may include non-specific names, grouped assertions, misdirected assertions, inter-test dependencies, lack of specificity, etc. These make maintenance or even basic understanding almost impossible without direct knowledge. Class and asset conventions are almost negligible in comparison which is why I do favour adding them as non-binding loose principles but don't consider them essential. I can relate to targeting the low hanging fruit (and there's NOTHING wrong with doing so) but it's only going to be one small step along a very long path if our objective is to improve maintenance.

>> When testing, I want to be agile: what is the behavior I'm testing, and
>> how do I express that in code succinctly? If I can simply embed another
>> class in the same file, or even another namespace, that's often faster
>
>Sidenote: what might be succinct for one developer (using what they'd consider to be obvious
>placements and/or practices) might not be as apparent to another.

This is true, but overall I still think they're minor. In-file classes aren't that uncommon or difficult to find (though I will admit using autoloading may make it harder - there's no automatic assumption possible that a non-required class must be inline). Assets need to be dragged in predictable ways, and autoloading vs requiring/including just trades automagic convenience for explicitness (preferable IMO when possible).

>> I think some loose guidelines may be in order, but I don't think they
>> should be hard directives; if they get in the way of efficiently
>> creating good tests, they should be ignored.
>
>I like the term loose guidelines. I'd say ideally if there are LOTS of assets being created, they
>should generally go into a TestAsset namespace.
>
>Ideally, I think putting them inside the same file as the test should be a no-go though. In most
>developers heads already is the concept that there is a 1-1 mapping of class to file.  If a
>developer ever wanted track down a reference, with the 1-1 rule, it's clear where their first
>look would be.  If it's not there, currently you have to go poking through all of the components >code to try and identify the proper source.
>
>>> It should be noted my opinion is just my opinion. I also believe code
>>> coverage is a complete waste of time, our tests too complicated and
>>> interrelated, and we should all convert to worshipping the Spaghetti
>>> Monster ;).
>
>Interesting point.. my opinion here is based on the fact that ZF is developed and maintained by a
>community of people, not just a few core developers.  Code coverage is a hard and fast rule that
>forces people to get to some level of conformance.  Conformance is good when it comes to getting
>new developers up to speed.  This is also the same driving factor that drives the opinion that
>keeping assets in a known place will make it easier for new developers to just "jump in" and
>start contributing.

Problem, in my mind, is that conformance is currently not reviewed beyond code coverage so essentially the measure is meaningless. There are more than a few test suites in ZF that have a high code coverage but on further inspection are little more than spoofed tests that lead nowhere. On the other hand, you also get the poor sods (I have to be symphathetic to myself!) who end up with a code coverage of 75% with (I hope) good test coverage who struggle to make up the other 5-10% by adding additional tests we may view as meaningless (testing getters/setters - meh).

I suppose that's largely why I tend to criticise code coverage, it's only useful when examined in some overall context, and nobody has actually stated what that context is. All that does is encourage developers who may be less than stellar testers to articifically inflate the measurement without consequences (like finding Matthew at their doorstep one day with a shillelagh and a frown ;)).

Paddy
 
Pádraic Brady

http://blog.astrumfutura.com
http://www.survivethedeepend.com
OpenID Europe Foundation Irish Representative



From: Ralph Schindler <[hidden email]>
To: [hidden email]
Sent: Wed, April 14, 2010 6:44:22 PM
Subject: Re: [zf-contributors] Testing best practices opinion needed on testing assets


>> If anything, I'd prefer we focus more on quality than cleanliness. Cleanliness
>> is meaningless without quality IMO. I'd rate the importance of a proper test
>> name quite a bit above whether or not the test used an in-file or namespaced
>> class that took the tester extra time to separate.

I agree with you here.  The basis of the argument is not for quality (which is important) but for maintenance.  If there is a "loose" rule of where to expect to find assets, then *other* developers that want to contribute will have an expectation of where to find things when digging into the tests.

> When testing, I want to be agile: what is the behavior I'm testing, and
> how do I express that in code succinctly? If I can simply embed another
> class in the same file, or even another namespace, that's often faster

Sidenote: what might be succinct for one developer (using what they'd consider to be obvious placements and/or practices) might not be as apparent to another.

> I think some loose guidelines may be in order, but I don't think they
> should be hard directives; if they get in the way of efficiently
> creating good tests, they should be ignored.

I like the term loose guidelines. I'd say ideally if there are LOTS of assets being created, they should generally go into a TestAsset namespace.

Ideally, I think putting them inside the same file as the test should be a no-go though. In most developers heads already is the concept that there is a 1-1 mapping of class to file.  If a developer ever wanted track down a reference, with the 1-1 rule, it's clear where their first look would be.  If it's not there, currently you have to go poking through all of the components code to try and identify the proper source.

>> It should be noted my opinion is just my opinion. I also believe code
>> coverage is a complete waste of time, our tests too complicated and
>> interrelated, and we should all convert to worshipping the Spaghetti
>> Monster ;).

Interesting point.. my opinion here is based on the fact that ZF is developed and maintained by a community of people, not just a few core developers.  Code coverage is a hard and fast rule that forces people to get to some level of conformance.  Conformance is good when it comes to getting new developers up to speed.  This is also the same driving factor that drives the opinion that keeping assets in a known place will make it easier for new developers to just "jump in" and start contributing.

-ralph
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

till
In reply to this post by Ralph Schindler-2
On Wed, Apr 14, 2010 at 7:58 PM, Ralph Schindler
<[hidden email]> wrote:

>
>> On require_once:
>>
>> I think stripping them from the framework is a good idea, but on the
>> test suite -- isn't it also overcomplicating things?
>> I mean, for me stripping require_once calls is primarily a performance
>> thing. I've been doing it in production code for over a year or so
>> now. But I've never done it inside my test suite. Part of this is --
>> it doesn't really matter how many milliseconds I gain here, especially
>> not at the expense of more sophisticated development and "best
>> practices".
>>
>> I'd rather prefer slow straight-forward tests.
>
> It's actually not about performance at all, it's about the ability to find
> things where one might expect them to be.  This, above all else, speaks
> towards maintenance and ability for new developers to understand where
> things are expected to go, and where current things are expected to be.
>
> This is also supported by future infrastructure.. It is (will be) easy to
> have everything loadable by SPLClassLoader.
>
> http://wiki.php.net/rfc/splclassloader

I don't know -- as I said -- in the framework code - why not. But in
tests? Adds a lot on top for developers to "learn" in order to
contribute. On the other hand, most (if not all) are already familiar
with require_once.

I see your point, but I don't see this maintenance issue.

Not everything has to be standardized. Personally, I'm happy if there
are tests at all. The issues you want to solve should be looked at
further down the road when there are too many tests or something like
that. ;-) [Spoiler: This will never happen.]

Right now, I'd really like decent coverage on all components. And when
I say coverage, I don't necessarily mean that phpunit reports a green
class file. This would be my primary objective, which implies making
it really easy to contribute. :-)


Good stuff though -- this discussion/thread has a lot of insight,
thanks for starting it!

Till
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

drm-4
In reply to this post by Ralph Schindler-2
On 4/13/2010 8:22 PM, Ralph Schindler wrote:
> What do you think?

I'm just dropping my 0,02 cents (I know, that's 2 hundredths of a cent,
go figure...:P) here. I don't have much experience on a practical level
maintaining components, I am duly aware of that, so forgive me if this
is just too darn utopian.

In my opinion, (and in an ideal world ...;)) the writer of the component
would provide mock and dummy classes to make testing easier. I feel
test-driven development is more than writing your tests first, I think
it should provide insight to users of components how they can utilize
the tests for their own needs, and as such they actually become more of
a tool for all developers than just for the maintainer of the component.

In that light, it would make more sense to provide dummy and mock
classes alongside the actual implementations. For example, I recently
implemented a mailtransport implementation which simply holds the sent
emails in an array, so you can test for sending mails. It makes sense to
provide this is a Transport alongside the Sendmail and SMTP transport as
a Mock transport, since it will not only be used by tests for Zend_Mail
components, but also by any other test that somehow relates to sending
mails, without wanting to actually deliver them (application level tests).

I think if you can somehow relate to that part of the development from a
ZF-user perspective, people might actually get more familiar with
testing (it might just seem like something "hard", or is basically
unknown by users), and as such get them more comfortable providing tests
for code they contribute.

To sum up: I think you (we) should start moving all mock implementations
into the core of the component, and make them part of it. Any other
asset to the tests is just a fixture like any other piece of data.

I know it is a long shot, but 2.0 is all about cleaning out the closet,
right ...? ;-)


Gerard
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

Giorgio Sironi
On Fri, Apr 16, 2010 at 10:45 AM, drm <[hidden email]> wrote:
> To sum up: I think you (we) should start moving all mock implementations
> into the core of the component, and make them part of it. Any other asset to
> the tests is just a fixture like any other piece of data.

Usually mocks/stubs are automatically generated, so there's nothing to
move in the lib/ folder. :) For the hand-rolled stubs, I don't think
it's very useful include classes which are not necessary to run the
component...

--
Giorgio Sironi
Piccolo Principe & Web Engineer
http://giorgiosironi.blogspot.com
http://twitter.com/giorgiosironi
Reply | Threaded
Open this post in threaded view
|

Re: Testing best practices opinion needed on testing assets

padraicb
I think Gerard was referring more to dummy classes which fake interactions with external systems like an email server, database, file system, web server. In that case, I think we already have most of these and they do form part of the component library to allow users employ them in their own tests where the related components are in use.

Paddy
 
Pádraic Brady

http://blog.astrumfutura.com
http://www.survivethedeepend.com
OpenID Europe Foundation Irish Representative



From: Giorgio Sironi <[hidden email]>
To: drm <[hidden email]>
Cc: [hidden email]
Sent: Fri, April 16, 2010 11:39:03 AM
Subject: Re: [zf-contributors] Testing best practices opinion needed on testing assets

On Fri, Apr 16, 2010 at 10:45 AM, drm <[hidden email]> wrote:
> To sum up: I think you (we) should start moving all mock implementations
> into the core of the component, and make them part of it. Any other asset to
> the tests is just a fixture like any other piece of data.

Usually mocks/stubs are automatically generated, so there's nothing to
move in the lib/ folder. :) For the hand-rolled stubs, I don't think
it's very useful include classes which are not necessary to run the
component...

--
Giorgio Sironi
Piccolo Principe & Web Engineer
http://giorgiosironi.blogspot.com
http://twitter.com/giorgiosironi
12