Are simpletests OK?

[Jaroslav Tulach]

Hi.
I'd like to finish the necessary work on
http://wiki.netbeans.org/FitnessTestsWithoutX 
next week and integrate that to trunk. So far I am not aware of any unfixable
blocker, but I thought, I'd rather ask here. Testing fans, are you fine with
the changes done so far? Is there anything that has to be done before merge?
Is there anything you'd like to see by end of 6.5 release?

Thanks.
-jst

[tomwheel]

I have the following feedback, although these are mostly from me
trying it out last month and may no longer be valid. I am fetching a
new copy today and hope to have time to try it out again.  Only one
problem (#136621, described below) I have found so far would prevent
me from upgrading to 6.5.

There is a document called XTestReplacementCookBook linked to from here:
    http://wiki.netbeans.org/FitnessTestsWithoutX

but it is on a non-public server.  It seems to me that this would
possibly be useful documentation to help others transition away from
XTest and it would be nice if a copy of the document could be posted
on a public server.

Not necessarily a problem with simpletests itself, but I currently
cannot run functional tests on a platform application using Jellytools
-- I have to include several unwanted clusters (such as nb, ide,
apisupport, harness and java) in order for it to work.  Since XTest
did not use modules, this was not a problem under XTest:

    http://www.netbeans.org/issues/show_bug.cgi?id=136621

 - Removing a module from the suite does not remove it from the list
of functional test dependencies for a given module.

 - Adding additional test types (e.g. "performance") is possible, but
it seemed more difficult than it could be.

 - The new test infrastructure did not appear to kill functional tests
which exceed a reasonable timeout, as XTest did.

--
Tom Wheeler
http://www.tomwheeler.com/

[Jaroslav Tulach]

Hello Tom,
I was afraid to get more negative comments from you. Thanks for making me
happier. Anyway I am CCing Shura, as a lot of your comments are targeted to
him.

Dne Monday 23 June 2008 17:18:41 Tom Wheeler napsal(a):
> I have the following feedback, although these are mostly from me
> trying it out last month and may no longer be valid. I am fetching a
> new copy today and hope to have time to try it out again.  Only one
> problem (#136621, described below) I have found so far would prevent
> me from upgrading to 6.5.
>
> There is a document called XTestReplacementCookBook linked to from here:
>     http://wiki.netbeans.org/FitnessTestsWithoutX

Shura, do you think the document contains useful enough info to be moved to
public server?

> but it is on a non-public server.  It seems to me that this would
> possibly be useful documentation to help others transition away from
> XTest and it would be nice if a copy of the document could be posted
> on a public server.
>
> Not necessarily a problem with simpletests itself, but I currently
> cannot run functional tests on a platform application using Jellytools
> -- I have to include several unwanted clusters (such as nb, ide,
> apisupport, harness and java) in order for it to work.  Since XTest
> did not use modules, this was not a problem under XTest:
>
>     http://www.netbeans.org/issues/show_bug.cgi?id=136621

Again, a question to Shura. Will you split the Jellytools module to be more
modular? Btw. this bug shall now be part of product quality criteria, as
Jellytools will be in nb6.x cluster...

>  - Removing a module from the suite does not remove it from the list
> of functional test dependencies for a given module.

Does that work for unit dependencies? If so, please report that against
apisupport.project...

>  - Adding additional test types (e.g. "performance") is possible, but
> it seemed more difficult than it could be.

Actually, it is not supported. We do not have a need for that. But if you know
what changes to make in common build scripts, we can indeed accept your
patches.

>  - The new test infrastructure did not appear to kill functional tests
> which exceed a reasonable timeout, as XTest did.

Here we rely on Ant's JUnit and its timeout (10min by default). I've seen some
tests killed by exceeding 10min execution time, so basically the
functionality works. You need to provide more details to help us or Ant guys
reproduce the problem.

-jst

[shura-4]

On Jun 24, 2008, at 11:03 AM, Jaroslav Tulach wrote:

> Hello Tom,
> I was afraid to get more negative comments from you. Thanks for  
> making me
> happier. Anyway I am CCing Shura, as a lot of your comments are  
> targeted to
> him.
>
> Dne Monday 23 June 2008 17:18:41 Tom Wheeler napsal(a):
>> I have the following feedback, although these are mostly from me
>> trying it out last month and may no longer be valid. I am fetching a
>> new copy today and hope to have time to try it out again.  Only one
>> problem (#136621, described below) I have found so far would prevent
>> me from upgrading to 6.5.
>>
>> There is a document called XTestReplacementCookBook linked to from  
>> here:
>>    http://wiki.netbeans.org/FitnessTestsWithoutX
>
> Shura, do you think the document contains useful enough info to be  
> moved to
> public server?

Yes, I could move it to the public wiki.

Will do.

>
>
>> but it is on a non-public server.  It seems to me that this would
>> possibly be useful documentation to help others transition away from
>> XTest and it would be nice if a copy of the document could be posted
>> on a public server.
>>
>> Not necessarily a problem with simpletests itself, but I currently
>> cannot run functional tests on a platform application using  
>> Jellytools
>> -- I have to include several unwanted clusters (such as nb, ide,
>> apisupport, harness and java) in order for it to work.  Since XTest
>> did not use modules, this was not a problem under XTest:
>>
>>    http://www.netbeans.org/issues/show_bug.cgi?id=136621
>
> Again, a question to Shura. Will you split the Jellytools module to  
> be more
> modular?

I will start on it today.

Shura.

> Btw. this bug shall now be part of product quality criteria, as
> Jellytools will be in nb6.x cluster...
>
>
>> - Removing a module from the suite does not remove it from the list
>> of functional test dependencies for a given module.
>
> Does that work for unit dependencies? If so, please report that  
> against
> apisupport.project...
>
>> - Adding additional test types (e.g. "performance") is possible, but
>> it seemed more difficult than it could be.
>
> Actually, it is not supported. We do not have a need for that. But  
> if you know
> what changes to make in common build scripts, we can indeed accept  
> your
> patches.
>
>> - The new test infrastructure did not appear to kill functional tests
>> which exceed a reasonable timeout, as XTest did.
>
> Here we rely on Ant's JUnit and its timeout (10min by default). I've  
> seen some
> tests killed by exceeding 10min execution time, so basically the
> functionality works. You need to provide more details to help us or  
> Ant guys
> reproduce the problem.
>
> -jst
>