Does JUnit 4 now support dependencies?

[Hans Schwaebli]

I now think that test dependencies are a good idea.

Does JUnit 4.x support such dependencies, or if not, is it planned for
a certain JUnit release to be added?

For example, if the login test did not work, than all other tests
cannot work. They depend on the login test. If login test fails, the
other tests are not executed.

[kentb]

Hans,
 
I agree that test dependencies are a good idea. In my experience, though,
they aren't worth the investment to express them explicitly. I would rather
put the same time into improving the design of the software so the tests can
run fast.
 
I wonder if it would be possible to infer dependencies from historical data,
though. That and the "most recently broken first" heuristic might provide
nearly optimal "time to first failure", which I assume is your goal in
wishing for dependencies.
 
Regards,
 
Kent Beck
Three Rivers Institute

  _____  

From: junit@... [mailto:junit@...] On Behalf Of
hans_schwaebli
Sent: Wednesday, July 16, 2008 5:58 AM
To: junit@...
Subject: [junit] Does JUnit 4 now support dependencies?



I now think that test dependencies are a good idea.

Does JUnit 4.x support such dependencies, or if not, is it planned for
a certain JUnit release to be added?

For example, if the login test did not work, than all other tests
cannot work. They depend on the login test. If login test fails, the
other tests are not executed.



 


[Non-text portions of this message have been removed]

[Hans Schwaebli]

Hi Kent,

specifying dependencies is a little extra work, I agree. But it has advantages.

Consecutive tests which depend on failed tests, are not executed, which speeds up the tests. It also increases the information value. People are not confused with dozens or even hundreds of failed tests when just one or two tests really failed on which most other tests depend. I know of situations where there are dozenzs of failed tests (or more) reported by CruiseControl, and noone looks into them until one week before release date. This happens. It makes developers afraid to look into the apparent mess. If there would be just one or two failed tests, they would look into it more likely, and sooner.

The heuristic strategy would not really solve the problem. since it is not equivalent to the dependency specification. It would still run all tests, but the likelyhood that the first run tests are the cause for all the errors would not be very high I think, whereas you could trust the dependencies very much if they are set correctly.

Dependencies are a frequent topic. We have it in OSGI, in issue tracking software, in project management software, in requirement specifications and in test specifications. Why not support it in JUnit tests too?

Well, this is just some quick thoughts about it. This issue alone will certainly not draw people to other test frameworks, but it would improve JUnit I think. Noone would be forced to use that feature, but I think it would be good, if one could use it with JUnit, even in an coarse way (for example if the dependency "testApplicationServerRuns" passed).

Bye,

Hans


--- On Thu, 7/17/08, Kent Beck <kentb@...> wrote:
From: Kent Beck <kentb@...>
Subject: RE: [junit] Does JUnit 4 now support dependencies?
To: junit@...
Date: Thursday, July 17, 2008, 9:28 AM










   
            Hans,

 

I agree that test dependencies are a good idea. In my experience, though,

they aren't worth the investment to express them explicitly. I would rather

put the same time into improving the design of the software so the tests can

run fast.

 

I wonder if it would be possible to infer dependencies from historical data,

though. That and the "most recently broken first" heuristic might provide

nearly optimal "time to first failure", which I assume is your goal in

wishing for dependencies.

 

Regards,

 

Kent Beck

Three Rivers Institute



_____  



From: junit@yahoogroups. com [mailto:junit@yahoogroups. com] On Behalf Of

hans_schwaebli

Sent: Wednesday, July 16, 2008 5:58 AM

To: junit@yahoogroups. com

Subject: [junit] Does JUnit 4 now support dependencies?



I now think that test dependencies are a good idea.



Does JUnit 4.x support such dependencies, or if not, is it planned for

a certain JUnit release to be added?



For example, if the login test did not work, than all other tests

cannot work. They depend on the login test. If login test fails, the

other tests are not executed.



[Non-text portions of this message have been removed]




     

   
   
       
         
       
       








       


       
       


     

[Non-text portions of this message have been removed]

[Daniel Brolund-2]

Hi all,

I kind of agree with Hans, and I have another viewpoint on this
dependency thing:

JUnit is already a set of "execution blocks" with "implicit"
dependencies, e.g. a @Test method depends on the @Before method, @After
depends on [@Before, @Test], [@Before, @Test, @After] depends on
@BeforeClass etc.

If we would make those explicit by some mechanism/abstraction, I'm not
sure exactly how, we could probably use the same mechanism/abstraction
to declare dependencies.
If we could expose this dependency tree (unidirectional graph?), it
might be possible to add behaviors, like parallel execution, exclusion
of dependent tests on failure, etc.
Does this make sense?

Just my $0.0031416.

Cheers
Daniel

PS The reason I would like these features in JUnit is that JUnit is a
defacto standard (available in virtually all build systems, IDEs etc)
and almost every developer I know can run the JUnit tests in their IDE.
The threshold is very low.
But, when faced with e.g. long(er) running Selenium tests, you want to
parallelize, and you don't want to waste CPU/gridnodes on unneccessary
tests. Yes, I know it is supposed to be a unit test framework... ;-)


On Thu, 2008-07-17 at 08:58 -0700, Hans Schwaebli wrote:

> Hi Kent,
>
> specifying dependencies is a little extra work, I agree. But it has
> advantages.
>
> Consecutive tests which depend on failed tests, are not executed,
> which speeds up the tests. It also increases the information value.
> People are not confused with dozens or even hundreds of failed tests
> when just one or two tests really failed on which most other tests
> depend. I know of situations where there are dozenzs of failed tests
> (or more) reported by CruiseControl, and noone looks into them until
> one week before release date. This happens. It makes developers afraid
> to look into the apparent mess. If there would be just one or two
> failed tests, they would look into it more likely, and sooner.
>
> The heuristic strategy would not really solve the problem. since it is
> not equivalent to the dependency specification. It would still run all
> tests, but the likelyhood that the first run tests are the cause for
> all the errors would not be very high I think, whereas you could trust
> the dependencies very much if they are set correctly.
>
> Dependencies are a frequent topic. We have it in OSGI, in issue
> tracking software, in project management software, in requirement
> specifications and in test specifications. Why not support it in JUnit
> tests too?
>
> Well, this is just some quick thoughts about it. This issue alone will
> certainly not draw people to other test frameworks, but it would
> improve JUnit I think. Noone would be forced to use that feature, but
> I think it would be good, if one could use it with JUnit, even in an
> coarse way (for example if the dependency "testApplicationServerRuns"
> passed).
>
> Bye,
>
> Hans
>
> --- On Thu, 7/17/08, Kent Beck <kentb@...> wrote:
> From: Kent Beck <kentb@...>
> Subject: RE: [junit] Does JUnit 4 now support dependencies?
> To: junit@...
> Date: Thursday, July 17, 2008, 9:28 AM
>
> Hans,
>
> I agree that test dependencies are a good idea. In my experience,
> though,
>
> they aren't worth the investment to express them explicitly. I would
> rather
>
> put the same time into improving the design of the software so the
> tests can
>
> run fast.
>
> I wonder if it would be possible to infer dependencies from historical
> data,
>
> though. That and the "most recently broken first" heuristic might
> provide
>
> nearly optimal "time to first failure", which I assume is your goal in
>
> wishing for dependencies.
>
> Regards,
>
> Kent Beck
>
> Three Rivers Institute
>
> _____
>
> From: junit@yahoogroups. com [mailto:junit@yahoogroups. com] On Behalf
> Of
>
> hans_schwaebli
>
> Sent: Wednesday, July 16, 2008 5:58 AM
>
> To: junit@yahoogroups. com
>
> Subject: [junit] Does JUnit 4 now support dependencies?
>
> I now think that test dependencies are a good idea.
>
> Does JUnit 4.x support such dependencies, or if not, is it planned for
>
> a certain JUnit release to be added?
>
> For example, if the login test did not work, than all other tests
>
> cannot work. They depend on the login test. If login test fails, the
>
> other tests are not executed.
>
> [Non-text portions of this message have been removed]
>
>
>
>
>
>
>
>
>
>
>
> [Non-text portions of this message have been removed]
>
>
>
>
>  

[David Saff]

Hans,

There is a way of getting very close to what you want in JUnit 4.4 and
beyond, with assumptions:

private boolean canLogin() {
  // return true if login works
}

@Test public void login() {
   assertTrue(canLogin());
}

@Test public void launchMissiles() {
   assumeTrue(canLogin());
   launchMissiles();
   assertFalse(enemyBase().exists());
}

This is more general than test dependencies, because it allows tests
to depend on conditions that are not really tested (for example, you
might have tests that assume you're running on Windows).  Will that
work for you?

   David Saff

On Thu, Jul 17, 2008 at 11:58 AM, Hans Schwaebli
<hans_schwaebli@...> wrote:

> Hi Kent,
>
> specifying dependencies is a little extra work, I agree. But it has
> advantages.
>
> Consecutive tests which depend on failed tests, are not executed, which
> speeds up the tests. It also increases the information value. People are not
> confused with dozens or even hundreds of failed tests when just one or two
> tests really failed on which most other tests depend. I know of situations
> where there are dozenzs of failed tests (or more) reported by CruiseControl,
> and noone looks into them until one week before release date. This happens.
> It makes developers afraid to look into the apparent mess. If there would be
> just one or two failed tests, they would look into it more likely, and
> sooner.
>
> The heuristic strategy would not really solve the problem. since it is not
> equivalent to the dependency specification. It would still run all tests,
> but the likelyhood that the first run tests are the cause for all the errors
> would not be very high I think, whereas you could trust the dependencies
> very much if they are set correctly.
>
> Dependencies are a frequent topic. We have it in OSGI, in issue tracking
> software, in project management software, in requirement specifications and
> in test specifications. Why not support it in JUnit tests too?
>
> Well, this is just some quick thoughts about it. This issue alone will
> certainly not draw people to other test frameworks, but it would improve
> JUnit I think. Noone would be forced to use that feature, but I think it
> would be good, if one could use it with JUnit, even in an coarse way (for
> example if the dependency "testApplicationServerRuns" passed).
>
> Bye,
>
> Hans
>
> --- On Thu, 7/17/08, Kent Beck <kentb@...> wrote:
> From: Kent Beck <kentb@...>
> Subject: RE: [junit] Does JUnit 4 now support dependencies?
> To: junit@...
> Date: Thursday, July 17, 2008, 9:28 AM
>
> Hans,
>
> I agree that test dependencies are a good idea. In my experience, though,
>
> they aren't worth the investment to express them explicitly. I would rather
>
> put the same time into improving the design of the software so the tests can
>
> run fast.
>
> I wonder if it would be possible to infer dependencies from historical data,
>
> though. That and the "most recently broken first" heuristic might provide
>
> nearly optimal "time to first failure", which I assume is your goal in
>
> wishing for dependencies.
>
> Regards,
>
> Kent Beck
>
> Three Rivers Institute
>
> _____
>
> From: junit@yahoogroups. com [mailto:junit@yahoogroups. com] On Behalf Of
>
> hans_schwaebli
>
> Sent: Wednesday, July 16, 2008 5:58 AM
>
> To: junit@yahoogroups. com
>
> Subject: [junit] Does JUnit 4 now support dependencies?
>
> I now think that test dependencies are a good idea.
>
> Does JUnit 4.x support such dependencies, or if not, is it planned for
>
> a certain JUnit release to be added?
>
> For example, if the login test did not work, than all other tests
>
> cannot work. They depend on the login test. If login test fails, the
>
> other tests are not executed.
>
> [Non-text portions of this message have been removed]
>
>
>
>
>
>
>
>
>
>
>
> [Non-text portions of this message have been removed]
>
>

[adrian.kuhn]

Dear Hans,

have a look at JExample, it extends JUnit with first-class dependencies. You can declare
@Depends(...) which is taken into account when running the test graph. Tests may have
return values even, which are injected as parameter to dependent tests.

http://www.iam.unibe.ch/~scg/Research/JExample/

cheers,
Adrian


--- In junit@..., "hans_schwaebli" <hans_schwaebli@...> wrote:

>
> I now think that test dependencies are a good idea.
>
> Does JUnit 4.x support such dependencies, or if not, is it planned for
> a certain JUnit release to be added?
>
> For example, if the login test did not work, than all other tests
> cannot work. They depend on the login test. If login test fails, the
> other tests are not executed.
>

[toalexsmail]

Hi, David!
I just want to understand you point. Suppose I have 200 test method in
50 classes. In this class every test method should be somethiing like
this:

@Test
public void launchMissiles() {
   assumeTrue(A1.canLogin());
   assumeTrue(A1.canSomethingElse());
...
   assumeTrue(A50.canSomethingCompletelyDifferent());

    launchMissiles();
    assertFalse(enemyBase().exists());
}

--- In junit@..., "David Saff" <david@...> wrote:

>
> Hans,
>
> There is a way of getting very close to what you want in JUnit 4.4 and
> beyond, with assumptions:
>
> private boolean canLogin() {
>   // return true if login works
> }
>
> @Test public void login() {
>    assertTrue(canLogin());
> }
>
> @Test public void launchMissiles() {
>    assumeTrue(canLogin());
>    launchMissiles();
>    assertFalse(enemyBase().exists());
> }
>
> This is more general than test dependencies, because it allows tests
> to depend on conditions that are not really tested (for example, you
> might have tests that assume you're running on Windows).  Will that
> work for you?
>
>    David Saff
>
> On Thu, Jul 17, 2008 at 11:58 AM, Hans Schwaebli
> <hans_schwaebli@...> wrote:
> > Hi Kent,
> >
> > specifying dependencies is a little extra work, I agree. But it has
> > advantages.
> >
> > Consecutive tests which depend on failed tests, are not executed,

which
> > speeds up the tests. It also increases the information value.
People are not
> > confused with dozens or even hundreds of failed tests when just
one or two
> > tests really failed on which most other tests depend. I know of
situations
> > where there are dozenzs of failed tests (or more) reported by
CruiseControl,
> > and noone looks into them until one week before release date. This
happens.
> > It makes developers afraid to look into the apparent mess. If
there would be
> > just one or two failed tests, they would look into it more likely, and
> > sooner.
> >
> > The heuristic strategy would not really solve the problem. since
it is not
> > equivalent to the dependency specification. It would still run all
tests,
> > but the likelyhood that the first run tests are the cause for all
the errors
> > would not be very high I think, whereas you could trust the
dependencies
> > very much if they are set correctly.
> >
> > Dependencies are a frequent topic. We have it in OSGI, in issue
tracking
> > software, in project management software, in requirement
specifications and
> > in test specifications. Why not support it in JUnit tests too?
> >
> > Well, this is just some quick thoughts about it. This issue alone will
> > certainly not draw people to other test frameworks, but it would
improve
> > JUnit I think. Noone would be forced to use that feature, but I
think it
> > would be good, if one could use it with JUnit, even in an coarse
way (for

> > example if the dependency "testApplicationServerRuns" passed).
> >
> > Bye,
> >
> > Hans
> >
> > --- On Thu, 7/17/08, Kent Beck <kentb@...> wrote:
> > From: Kent Beck <kentb@...>
> > Subject: RE: [junit] Does JUnit 4 now support dependencies?
> > To: junit@...
> > Date: Thursday, July 17, 2008, 9:28 AM
> >
> > Hans,
> >
> > I agree that test dependencies are a good idea. In my experience,

though,
> >
> > they aren't worth the investment to express them explicitly. I
would rather
> >
> > put the same time into improving the design of the software so the
tests can
> >
> > run fast.
> >
> > I wonder if it would be possible to infer dependencies from
historical data,
> >
> > though. That and the "most recently broken first" heuristic might
provide

> >
> > nearly optimal "time to first failure", which I assume is your goal in
> >
> > wishing for dependencies.
> >
> > Regards,
> >
> > Kent Beck
> >
> > Three Rivers Institute
> >
> > _____
> >
> > From: junit@yahoogroups. com [mailto:junit@yahoogroups. com] On

Behalf Of

> >
> > hans_schwaebli
> >
> > Sent: Wednesday, July 16, 2008 5:58 AM
> >
> > To: junit@yahoogroups. com
> >
> > Subject: [junit] Does JUnit 4 now support dependencies?
> >
> > I now think that test dependencies are a good idea.
> >
> > Does JUnit 4.x support such dependencies, or if not, is it planned for
> >
> > a certain JUnit release to be added?
> >
> > For example, if the login test did not work, than all other tests
> >
> > cannot work. They depend on the login test. If login test fails, the
> >
> > other tests are not executed.
> >
> > [Non-text portions of this message have been removed]
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > [Non-text portions of this message have been removed]
> >
> >
>

[Hans Schwaebli]

Thanks for the suggestion. I didn't know this exists. Still I think, that dependencies should be supported by JUnit out of the box.


--- On Fri, 7/25/08, adrian.kuhn <adrian.kuhn@...> wrote:
From: adrian.kuhn <adrian.kuhn@...>
Subject: [junit] Re: Does JUnit 4 now support dependencies?
To: junit@...
Date: Friday, July 25, 2008, 7:11 AM










   
            Dear Hans,



have a look at JExample, it extends JUnit with first-class dependencies. You can declare

@Depends(... ) which is taken into account when running the test graph. Tests may have

return values even, which are injected as parameter to dependent tests.



http://www.iam. unibe.ch/ ~scg/Research/ JExample/



cheers,

Adrian



--- In junit@yahoogroups. com, "hans_schwaebli" <hans_schwaebli@ ...> wrote:

>

> I now think that test dependencies are a good idea.

>

> Does JUnit 4.x support such dependencies, or if not, is it planned for

> a certain JUnit release to be added?

>

> For example, if the login test did not work, than all other tests

> cannot work. They depend on the login test. If login test fails, the

> other tests are not executed.

>




     

   
   
       
         
       
       








       


       
       


     

[Non-text portions of this message have been removed]

[Hans Schwaebli]

I would see this as a workaround.

As I said, dependencies are a frequent and often important issue in organisation of various things (lets be general). So a direct support by JUnit for dependencies out of the box would be ideal I think.

In your example I would need to store the result of canLogin () into a global variable. Otherwise the canLogin-test would be executed multiple times if canLogin() dependency is needed in more than one test. This is what a framework like JUnit could do more productively and correctly than a programmer.

In test specifications often dependencies are stated (test Y depends on test X for example). A test implementer could simply use that information easily if JUnit would support dependencies out of the box, for example with a @Dependency annotation. Of course one would need to take care not to have cyclic test dependencies. This should be analysed before test execution by the framework.


--- On Wed, 7/23/08, David Saff <david@...> wrote:
From: David Saff <david@...>
Subject: Re: [junit] Does JUnit 4 now support dependencies?
To: junit@...
Date: Wednesday, July 23, 2008, 1:32 PM










   
            Hans,



There is a way of getting very close to what you want in JUnit 4.4 and

beyond, with assumptions:



private boolean canLogin() {

  // return true if login works

}



@Test public void login() {

   assertTrue(canLogin ());

}



@Test public void launchMissiles( ) {

   assumeTrue(canLogin ());

   launchMissiles( );

   assertFalse( enemyBase( ).exists( ));

}



This is more general than test dependencies, because it allows tests

to depend on conditions that are not really tested (for example, you

might have tests that assume you're running on Windows).  Will that

work for you?



David Saff



On Thu, Jul 17, 2008 at 11:58 AM, Hans Schwaebli

<hans_schwaebli@ yahoo.com> wrote:

> Hi Kent,

>

> specifying dependencies is a little extra work, I agree. But it has

> advantages.

>

> Consecutive tests which depend on failed tests, are not executed, which

> speeds up the tests. It also increases the information value. People are not

> confused with dozens or even hundreds of failed tests when just one or two

> tests really failed on which most other tests depend. I know of situations

> where there are dozenzs of failed tests (or more) reported by CruiseControl,

> and noone looks into them until one week before release date. This happens.

> It makes developers afraid to look into the apparent mess. If there would be

> just one or two failed tests, they would look into it more likely, and

> sooner.

>

> The heuristic strategy would not really solve the problem. since it is not

> equivalent to the dependency specification. It would still run all tests,

> but the likelyhood that the first run tests are the cause for all the errors

> would not be very high I think, whereas you could trust the dependencies

> very much if they are set correctly.

>

> Dependencies are a frequent topic. We have it in OSGI, in issue tracking

> software, in project management software, in requirement specifications and

> in test specifications. Why not support it in JUnit tests too?

>

> Well, this is just some quick thoughts about it. This issue alone will

> certainly not draw people to other test frameworks, but it would improve

> JUnit I think. Noone would be forced to use that feature, but I think it

> would be good, if one could use it with JUnit, even in an coarse way (for

> example if the dependency "testApplicationSer verRuns" passed).

>

> Bye,

>

> Hans

>

> --- On Thu, 7/17/08, Kent Beck <kentb@earthlink. net> wrote:

> From: Kent Beck <kentb@earthlink. net>

> Subject: RE: [junit] Does JUnit 4 now support dependencies?

> To: junit@yahoogroups. com

> Date: Thursday, July 17, 2008, 9:28 AM

>

> Hans,

>

> I agree that test dependencies are a good idea. In my experience, though,

>

> they aren't worth the investment to express them explicitly. I would rather

>

> put the same time into improving the design of the software so the tests can

>

> run fast.

>

> I wonder if it would be possible to infer dependencies from historical data,

>

> though. That and the "most recently broken first" heuristic might provide

>

> nearly optimal "time to first failure", which I assume is your goal in

>

> wishing for dependencies.

>

> Regards,

>

> Kent Beck

>

> Three Rivers Institute

>

> _____

>

> From: junit@yahoogroups. com [mailto:junit@ yahoogroups. com] On Behalf Of

>

> hans_schwaebli

>

> Sent: Wednesday, July 16, 2008 5:58 AM

>

> To: junit@yahoogroups. com

>

> Subject: [junit] Does JUnit 4 now support dependencies?

>

> I now think that test dependencies are a good idea.

>

> Does JUnit 4.x support such dependencies, or if not, is it planned for

>

> a certain JUnit release to be added?

>

> For example, if the login test did not work, than all other tests

>

> cannot work. They depend on the login test. If login test fails, the

>

> other tests are not executed.

>

> [Non-text portions of this message have been removed]

>

>

>

>

>

>

>

>

>

>

>

> [Non-text portions of this message have been removed]

>

>


     

   
   
       
         
       
       








       


       
       


     

[Non-text portions of this message have been removed]

[David Saff]

Daniel,

On Thu, Jul 17, 2008 at 3:39 PM, Daniel Brolund
<daniel.brolund@...> wrote:

> Hi all,
>
> I kind of agree with Hans, and I have another viewpoint on this
> dependency thing:
>
> JUnit is already a set of "execution blocks" with "implicit"
> dependencies, e.g. a @Test method depends on the @Before method, @After
> depends on [@Before, @Test], [@Before, @Test, @After] depends on
> @BeforeClass etc.
>
> If we would make those explicit by some mechanism/abstraction, I'm not
> sure exactly how, we could probably use the same mechanism/abstraction
> to declare dependencies.
> If we could expose this dependency tree (unidirectional graph?), it
> might be possible to add behaviors, like parallel execution, exclusion
> of dependent tests on failure, etc.
> Does this make sense?

Absolutely.  We've built a similar idea into BlockJUnit4ClassRunner in
JUnit 4.5.  Imagine:

public class RemoveFailingDependents extends BlockJUnit4ClassRunner {
        public RemoveFailingDependents(Class<?> klass) throws InitializationError {
                super(klass);
        }

        @Override
        protected void runChild(FrameworkMethod method, RunNotifier notifier) {
                for (FrameworkMethod each : getDependencies(method))
                        if (alreadyMarkedFailed(method)) {
                                notifier.fireTestIgnored(describeChild(method));
                                return;
                        }
                RunListener listener = new RunListener() {
                        @Override
                        public void testFailure(Failure failure) throws Exception {
                                markTestFailed(failure.getDescription());
                        }
                };
                notifier.addListener(listener);
                super.runChild(method, notifier);
                notifier.removeListener(listener);
        }
}

This leaves out the ordering that would need to be done by a sorter to
make sure that dependencies run before dependers.

   David Saff

[Daniel Brolund-2]

Hi David,

I think my other post explains in more detail what I'm after. :-)

Cheers
Daniel

On Thu, 2008-07-31 at 10:59 -0400, David Saff wrote:

> Daniel,
>
> On Thu, Jul 17, 2008 at 3:39 PM, Daniel Brolund
> <daniel.brolund@...> wrote:
> > Hi all,
> >
> > I kind of agree with Hans, and I have another viewpoint on this
> > dependency thing:
> >
> > JUnit is already a set of "execution blocks" with "implicit"
> > dependencies, e.g. a @Test method depends on the @Before method,
> @After
> > depends on [@Before, @Test], [@Before, @Test, @After] depends on
> > @BeforeClass etc.
> >
> > If we would make those explicit by some mechanism/abstraction, I'm
> not
> > sure exactly how, we could probably use the same
> mechanism/abstraction
> > to declare dependencies.
> > If we could expose this dependency tree (unidirectional graph?), it
> > might be possible to add behaviors, like parallel execution,
> exclusion
> > of dependent tests on failure, etc.
> > Does this make sense?
>
> Absolutely. We've built a similar idea into BlockJUnit4ClassRunner in
> JUnit 4.5. Imagine:
>
> public class RemoveFailingDependents extends BlockJUnit4ClassRunner {
> public RemoveFailingDependents(Class<?> klass) throws
> InitializationError {
> super(klass);
> }
>
> @Override
> protected void runChild(FrameworkMethod method, RunNotifier notifier)
> {
> for (FrameworkMethod each : getDependencies(method))
> if (alreadyMarkedFailed(method)) {
> notifier.fireTestIgnored(describeChild(method));
> return;
> }
> RunListener listener = new RunListener() {
> @Override
> public void testFailure(Failure failure) throws Exception {
> markTestFailed(failure.getDescription());
> }
> };
> notifier.addListener(listener);
> super.runChild(method, notifier);
> notifier.removeListener(listener);
> }
> }
>
> This leaves out the ordering that would need to be done by a sorter to
> make sure that dependencies run before dependers.
>
> David Saff
>
>
>
>  

[David Saff]

Alex,

First, assumptions can go in individual test methods, or in
@BeforeClass on a test class (to short-circuit many methods), or in
@BeforeClass on a suite (to short-circuit the entire suite).  Second,
I assume you'd use abstraction to eliminate duplication, just like you
do anywhere else.  The advantage of having explicit method calls as
assumptions is that you can use the abstraction mechanisms in Java to
group assumptions, rather than having to invent a new mechanism that
extracts sets of test description strings.

   David Saff

On Wed, Jul 30, 2008 at 4:54 PM, toalexsmail <toalexsmail@...> wrote:

> Hi, David!
> I just want to understand you point. Suppose I have 200 test method in
> 50 classes. In this class every test method should be somethiing like
> this:
>
> @Test
> public void launchMissiles() {
> assumeTrue(A1.canLogin());
> assumeTrue(A1.canSomethingElse());
> ...
> assumeTrue(A50.canSomethingCompletelyDifferent());
>
> launchMissiles();
> assertFalse(enemyBase().exists());
>
> }
>
> --- In junit@..., "David Saff" <david@...> wrote:
>>
>> Hans,
>>
>> There is a way of getting very close to what you want in JUnit 4.4 and
>> beyond, with assumptions:
>>
>> private boolean canLogin() {
>> // return true if login works
>> }
>>
>> @Test public void login() {
>> assertTrue(canLogin());
>> }
>>
>> @Test public void launchMissiles() {
>> assumeTrue(canLogin());
>> launchMissiles();
>> assertFalse(enemyBase().exists());
>> }
>>
>> This is more general than test dependencies, because it allows tests
>> to depend on conditions that are not really tested (for example, you
>> might have tests that assume you're running on Windows). Will that
>> work for you?
>>
>> David Saff
>>
>> On Thu, Jul 17, 2008 at 11:58 AM, Hans Schwaebli
>> <hans_schwaebli@...> wrote:
>> > Hi Kent,
>> >
>> > specifying dependencies is a little extra work, I agree. But it has
>> > advantages.
>> >
>> > Consecutive tests which depend on failed tests, are not executed,
> which
>> > speeds up the tests. It also increases the information value.
> People are not
>> > confused with dozens or even hundreds of failed tests when just
> one or two
>> > tests really failed on which most other tests depend. I know of
> situations
>> > where there are dozenzs of failed tests (or more) reported by
> CruiseControl,
>> > and noone looks into them until one week before release date. This
> happens.
>> > It makes developers afraid to look into the apparent mess. If
> there would be
>> > just one or two failed tests, they would look into it more likely, and
>> > sooner.
>> >
>> > The heuristic strategy would not really solve the problem. since
> it is not
>> > equivalent to the dependency specification. It would still run all
> tests,
>> > but the likelyhood that the first run tests are the cause for all
> the errors
>> > would not be very high I think, whereas you could trust the
> dependencies
>> > very much if they are set correctly.
>> >
>> > Dependencies are a frequent topic. We have it in OSGI, in issue
> tracking
>> > software, in project management software, in requirement
> specifications and
>> > in test specifications. Why not support it in JUnit tests too?
>> >
>> > Well, this is just some quick thoughts about it. This issue alone will
>> > certainly not draw people to other test frameworks, but it would
> improve
>> > JUnit I think. Noone would be forced to use that feature, but I
> think it
>> > would be good, if one could use it with JUnit, even in an coarse
> way (for
>> > example if the dependency "testApplicationServerRuns" passed).
>> >
>> > Bye,
>> >
>> > Hans
>> >
>> > --- On Thu, 7/17/08, Kent Beck <kentb@...> wrote:
>> > From: Kent Beck <kentb@...>
>> > Subject: RE: [junit] Does JUnit 4 now support dependencies?
>> > To: junit@...
>> > Date: Thursday, July 17, 2008, 9:28 AM
>> >
>> > Hans,
>> >
>> > I agree that test dependencies are a good idea. In my experience,
> though,
>> >
>> > they aren't worth the investment to express them explicitly. I
> would rather
>> >
>> > put the same time into improving the design of the software so the
> tests can
>> >
>> > run fast.
>> >
>> > I wonder if it would be possible to infer dependencies from
> historical data,
>> >
>> > though. That and the "most recently broken first" heuristic might
> provide
>> >
>> > nearly optimal "time to first failure", which I assume is your goal in
>> >
>> > wishing for dependencies.
>> >
>> > Regards,
>> >
>> > Kent Beck
>> >
>> > Three Rivers Institute
>> >
>> > _____
>> >
>> > From: junit@yahoogroups. com [mailto:junit@yahoogroups. com] On
> Behalf Of
>> >
>> > hans_schwaebli
>> >
>> > Sent: Wednesday, July 16, 2008 5:58 AM
>> >
>> > To: junit@yahoogroups. com
>> >
>> > Subject: [junit] Does JUnit 4 now support dependencies?
>> >
>> > I now think that test dependencies are a good idea.
>> >
>> > Does JUnit 4.x support such dependencies, or if not, is it planned for
>> >
>> > a certain JUnit release to be added?
>> >
>> > For example, if the login test did not work, than all other tests
>> >
>> > cannot work. They depend on the login test. If login test fails, the
>> >
>> > other tests are not executed.
>> >
>> > [Non-text portions of this message have been removed]
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > [Non-text portions of this message have been removed]
>> >
>> >
>>
>
>

[David Saff]

On Thu, Jul 31, 2008 at 3:46 AM, Hans Schwaebli
<hans_schwaebli@...> wrote:

> I would see this as a workaround.
>
> As I said, dependencies are a frequent and often important issue in
> organisation of various things (lets be general). So a direct support by
> JUnit for dependencies out of the box would be ideal I think.
>
> In your example I would need to store the result of canLogin () into a
> global variable. Otherwise the canLogin-test would be executed multiple
> times if canLogin() dependency is needed in more than one test. This is what
> a framework like JUnit could do more productively and correctly than a
> programmer.

Given that "boolean canLogin()" exists, I see a one-line difference between

public static boolean CAN_LOGIN = canLogin();
@Test public void myTest() {
  assumeTrue(CAN_LOGIN);

and

@DependsOn("nett.saff.myprogram.LoginTests.canLogin");
@Test public void myTest() {

For that one line, you get static checking and refactoring support
(did you notice the typo in "nett"?)

I'd love it if you could give the assumption framework a try, and
JExample, and report back your experiences.  I fear otherwise, we're
trading in hypotheticals.  Thanks,

   David

[Cédric Beust ♔]

This approach doesn't scale as soon as you need to depend on more than one
method (what do you do later down the road when you need to add one method
to the dependencies?  Update all the methods that depend on it?), which is
why I introduced groups in TestNG...

--
Cédric


On Fri, Aug 1, 2008 at 7:26 AM, David Saff <david@...> wrote:

>   On Thu, Jul 31, 2008 at 3:46 AM, Hans Schwaebli
>
> <hans_schwaebli@... <hans_schwaebli%40yahoo.com>> wrote:
> > I would see this as a workaround.
> >
> > As I said, dependencies are a frequent and often important issue in
> > organisation of various things (lets be general). So a direct support by
> > JUnit for dependencies out of the box would be ideal I think.
> >
> > In your example I would need to store the result of canLogin () into a
> > global variable. Otherwise the canLogin-test would be executed multiple
> > times if canLogin() dependency is needed in more than one test. This is
> what
> > a framework like JUnit could do more productively and correctly than a
> > programmer.
>
> Given that "boolean canLogin()" exists, I see a one-line difference between
>
> public static boolean CAN_LOGIN = canLogin();
> @Test public void myTest() {
> assumeTrue(CAN_LOGIN);
>
> and
>
> @DependsOn("nett.saff.myprogram.LoginTests.canLogin");
> @Test public void myTest() {
>
> For that one line, you get static checking and refactoring support
> (did you notice the typo in "nett"?)
>
> I'd love it if you could give the assumption framework a try, and
> JExample, and report back your experiences. I fear otherwise, we're
> trading in hypotheticals. Thanks,
>
> David
> &