[Fwd: Re: Aqsis shader compiler RTS]

[Matthias Baas-2]

Chris Foster wrote:
> The test suite obviously needs to run RSL files through the compiler and
> capture the output.  Some tests will be designed to cause compilation
> failure, so we need a way to specify "this test compiles" or "this test
> fails to compile".  

I agree. I already wanted to have that for aqsis as well (feeding it
malformed RIB and check how aqsis behaves).

> It would also be good to validate the output somehow
> for test cases which pass; the best method I can think for this is to
> run a text diff of the result with some reference slx files in much the
> same way as we currently run do image diffs.

Do you expect this to catch problems that would otherwise go unnoticed
by using the shader for rendering an image?
Is it already known in advance what the "assembly" code (or what it is
called?) for a given shader is supposed to look like? Aren't there many
valid possibilities how the code may look like? (and couldn't this
change when the compiler is modified so that it outputs more optimized
code or something like that?)
How do you create a correct reference slx in the first place? (an image
can be inspected and verified just by looking at it, but how do you do
that with an assembly listing?)

Well, in principle, dealing with tests that are only supposed to output
text (and an error code) instead of images is certainly more
straightforward. What do you think would be the ideal way to specify
those tests? If it is within the rendertester script, there could be a
new type of job in addition to the "RenderJob". Maybe just a
"CommandJob" that takes a command line that will be run and that also
specifies the expected return code, the expected contents of stdout and
stderr and a list of (text) files that are expected to be produced by
the command. Or maybe there could be a more specific "SLCompilerJob"
that is supposed to run a shading language compiler (maybe also from
other renderers, even though I don't know how the expected output can be
described independent from the actual compiler used).

- Matthias -

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[ajb-2]

G'day all.

Quoting Matthias Baas <mbaas@...>:

> Is it already known in advance what the "assembly" code (or what it is
> called?) for a given shader is supposed to look like? Aren't there many
> valid possibilities how the code may look like? (and couldn't this
> change when the compiler is modified so that it outputs more optimized
> code or something like that?)

I agree.  I think that, realistically, the easiest and most robust
thing to do is write a small test harness around the shadervm, and
use that for testing.

This has the advantage that you could change the renderer (e.g. the
pixel sampling algorithms) without breaking the shader tests.

Cheers,
Andrew Bromage

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Chris Foster]

Hi Matthias,

On Mon, Aug 04, 2008 at 11:37:40PM +0100, Matthias Baas wrote:
> > It would also be good to validate the output somehow
> > for test cases which pass; the best method I can think for this is to
> > run a text diff of the result with some reference slx files in much the
> > same way as we currently run do image diffs.
>
> Do you expect this to catch problems that would otherwise go unnoticed
> by using the shader for rendering an image?

No, probably not.  The image-based RTS is perfectly adequate for testing
some types of behaviour.  For those cases we can write tests for the
existing RTS.

> Is it already known in advance what the "assembly" code (or what it is
> called?) for a given shader is supposed to look like? Aren't there many
> valid possibilities how the code may look like?

Yes, but the point is more to *know* when things have changed rather
than test against some "one true output".  When writing unit tests, it's
inevitable that you need to refactor test cases when you refactor the
code itself.  Something similar applies here.

Our shader compiler is currently a very slowly moving target, so I doubt
maintaining the tests will be a huge burden.  Mostly it's had bugfixes
in the past two years - IIRC, the only major addition being
break/continue support and the associated refactoring.

> (and couldn't this change when the compiler is modified so that it
> outputs more optimized code or something like that?)

Yes, but that's a test of the optimization transformation.  If we
started doing significant optimizations we'd also have a flag to turn
that off.

It's possible that testing the shader "assembly" would actually be
*more* important if we had optimizations: a final image is unable to
indicate the effects of any code transformation (ie, speed).

> How do you create a correct reference slx in the first place? (an
> image can be inspected and verified just by looking at it, but how do
> you do that with an assembly listing?)

I'm more interested in knowing when the output *changes* rather than
comparing against some canonical "most correct" version.  Therefore we
simply record the output slx files from the current version of the
compiler to get the test cases.

I think this may be useful: sometimes I'll make a change to the shader
compiler and I don't expect the compiled output to change.  If I *do*
expect the output to change then I'll know in what way the generated
assembly should differ - then I can update the tests.

The tests I'm thinking of here are minimal cut-down test cases, so
understanding the generated assembly shouldn't be too much of an issue.

> Well, in principle, dealing with tests that are only supposed to output
> text (and an error code) instead of images is certainly more
> straightforward. What do you think would be the ideal way to specify
> those tests? If it is within the rendertester script, there could be a
> new type of job in addition to the "RenderJob". Maybe just a
> "CommandJob" that takes a command line that will be run and that also
> specifies the expected return code, the expected contents of stdout and
> stderr and a list of (text) files that are expected to be produced by
> the command. Or maybe there could be a more specific "SLCompilerJob"
> that is supposed to run a shading language compiler (maybe also from
> other renderers, even though I don't know how the expected output can be
> described independent from the actual compiler used).

I guess SLCompilerJob could be implemented as a wrapper around a more
general CommandJob type command?  A useful thing here would be to enable
us to use the existing renderer definitions for the locations of the
shader compiler executable.

~Chris.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Chris Foster]

On Mon, Aug 04, 2008 at 07:51:18PM -0400, ajb@... wrote:

> G'day all.
>
> Quoting Matthias Baas <mbaas@...>:
>
> > Is it already known in advance what the "assembly" code (or what it is
> > called?) for a given shader is supposed to look like? Aren't there many
> > valid possibilities how the code may look like? (and couldn't this
> > change when the compiler is modified so that it outputs more optimized
> > code or something like that?)
>
> I agree.  I think that, realistically, the easiest and most robust
> thing to do is write a small test harness around the shadervm, and
> use that for testing.

I'm not so interested in testing the shadervm here - for that we have
the image-based RTS, which I think does a passable job.  What I want is
a way to test the shader compiler itself; we currently have no good way
to do this.

> This has the advantage that you could change the renderer (e.g. the
> pixel sampling algorithms) without breaking the shader tests.

True, but I think a test harness for the shaderVM would be most valuable
for other reasons - in particular for performance testing.

Cheers,
~Chris.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Matthias Baas-2]

Hi,

I was just looking into adding those non-image-producing jobs to the
rendertester tool. Compiling a shader may then look like this:

SLCompilerJob(
        workingdir = "",
         shader = "test.sl",
         returnCode = -1,
         stdout = "",
         stderr =
"""08/11/2008 22:45:07 ERROR: test.sl : 4 : syntax error
08/11/2008 22:45:07 ERROR: Shader not compiled""",
        description = "Test job."
)

The expected return code, stdout and stderr text would then be compared
to the actual values. Now in the above case, this won't work because
aqsl outputs a timestamp before each line. How should we handle this?

- Allow something like "??/??/???? ??:??:?? ERROR: test.sl : 4 : syntax
error" where '?' may stand for any character? But then it must be
guaranteed that the size of each such unknown field is fixed.

- Allow regular expressions? This sounds like overkill to me (and makes
computing diffs not quite straightforward)

- Add an option to aqsl that prevents it from outputting a timestamp, so
that the expected output is always exactly known.

Thoughts?

- Matthias -


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Chris Foster]

On Mon, Aug 11, 2008 at 11:05:40PM +0100, Matthias Baas wrote:

> Hi,
>
> I was just looking into adding those non-image-producing jobs to the
> rendertester tool. Compiling a shader may then look like this:
>
> SLCompilerJob(
> workingdir = "",
>          shader = "test.sl",
>          returnCode = -1,
>          stdout = "",
>          stderr =
> """08/11/2008 22:45:07 ERROR: test.sl : 4 : syntax error
> 08/11/2008 22:45:07 ERROR: Shader not compiled""",
> description = "Test job."
> )
>
> The expected return code, stdout and stderr text would then be compared
> to the actual values.

Looks good :-)

> Now in the above case, this won't work because
> aqsl outputs a timestamp before each line. How should we handle this?
>
> - Allow something like "??/??/???? ??:??:?? ERROR: test.sl : 4 : syntax
> error" where '?' may stand for any character? But then it must be
> guaranteed that the size of each such unknown field is fixed.
>
> - Allow regular expressions? This sounds like overkill to me (and makes
> computing diffs not quite straightforward)
>
> - Add an option to aqsl that prevents it from outputting a timestamp, so
> that the expected output is always exactly known.
>
> Thoughts?

I don't like any of these options much.  The last one is ok-ish...

I think a better option would be to have a regex which strips the date
stamps from the start of each line *before* running the lines through
difflib.

We could implement the diff comparison operation between two strings as
a class with a compare(string1, string2) method or some such.  Then we'd
be free to specify various versions of the comparison operation, some
which modify the strings before performing the diff, some which just do
a plain diff etc.

~Chris.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Matthias Baas-3]

Hi,

I have just committed an initial version of this shader compiler testing
functionality. I think meanwhile the test tool is in a state where it
could do with a rewrite (it has been before but this was even a bigger
step away from a nice design). But well, maybe it still runs for a while.

There are now 2 new directories:

- ShaderCompilerTests: This contains the test files (similar to the RIBs
directory).

- referenceShaders: This is the equivalent to the "reference" directory
but contains compiled shaders (or other reference text files).

The main jobs.cfg file has been updated so that it also reads jobs from
the ShaderCompilerTests directory.
The file tasks.cfg has been extended with a task "rt_aqsl" that runs all
shader compiler tests (you can use the same options for only running a
subset of jobs as you can do with the other tasks).

I have created two example jobs that demonstrate the usage. One test
just compiles the plastic shader and checks its output whereas the other
test checks aqsl's behavior in the case of a shader that contains an
error (a missing semicolon).

Specifying a job basically works like mentioned in my previous mail. You
create a SLCompilerJob inside the config file that contains the shader
to compile and the expected result. You can check the return code,
stdout, stderr and any output files (by default, the compiled shader,
but it could be any text file). If you leave out any of the
"returnCode", "stdout", "stderr" argument, the respective output fill
not be checked. If a test doesn't produce a compiled shader (like the
above "missing colon" test), you have to set the "outputFiles" argument
to None or an empty list (otherwise the tool would expect the compiled
shader to appear).

The job internally uses regular expressions to remove those parts of the
output that differ from run to run (those parts are replaced by '*'
characters). In the case of stdout, it's the date stamp. On the compiled
shaders, it's the Aqsis version. Let me know if you find other things
that differ.
One somewhat weird behavior of aqsl is that in the case of an error
inside the shader it just mentions the base shader name in the error
message when the shader source begins within the first 2 lines of the
source file. If the shader begins later, aqsl prints the full absolute
path to the shader...?!? (which also makes it a bit more difficult to
compare error messages. So far, the tool doesn't handle this)

- Matthias -

Chris Foster wrote:

> On Mon, Aug 11, 2008 at 11:05:40PM +0100, Matthias Baas wrote:
>> Hi,
>>
>> I was just looking into adding those non-image-producing jobs to the
>> rendertester tool. Compiling a shader may then look like this:
>>
>> SLCompilerJob(
>> workingdir = "",
>>          shader = "test.sl",
>>          returnCode = -1,
>>          stdout = "",
>>          stderr =
>> """08/11/2008 22:45:07 ERROR: test.sl : 4 : syntax error
>> 08/11/2008 22:45:07 ERROR: Shader not compiled""",
>> description = "Test job."
>> )
>>
>> The expected return code, stdout and stderr text would then be compared
>> to the actual values.
>
> Looks good :-)
>
>> Now in the above case, this won't work because
>> aqsl outputs a timestamp before each line. How should we handle this?
>>
>> - Allow something like "??/??/???? ??:??:?? ERROR: test.sl : 4 : syntax
>> error" where '?' may stand for any character? But then it must be
>> guaranteed that the size of each such unknown field is fixed.
>>
>> - Allow regular expressions? This sounds like overkill to me (and makes
>> computing diffs not quite straightforward)
>>
>> - Add an option to aqsl that prevents it from outputting a timestamp, so
>> that the expected output is always exactly known.
>>
>> Thoughts?
>
> I don't like any of these options much.  The last one is ok-ish...
>
> I think a better option would be to have a regex which strips the date
> stamps from the start of each line *before* running the lines through
> difflib.
>
> We could implement the diff comparison operation between two strings as
> a class with a compare(string1, string2) method or some such.  Then we'd
> be free to specify various versions of the comparison operation, some
> which modify the strings before performing the diff, some which just do
> a plain diff etc.
>
> ~Chris.

>


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Chris Foster]

Hi Matthias,

On Mon, Aug 25, 2008 at 10:07:36AM +0100, Matthias Baas wrote:
> I have just committed an initial version of this shader compiler testing
> functionality.

Thanks!  I've wanted to test shaders for a long time, this will be
really nice.

> There are now 2 new directories:
>
> - ShaderCompilerTests: This contains the test files (similar to the RIBs
> directory).
>
> - referenceShaders: This is the equivalent to the "reference" directory
> but contains compiled shaders (or other reference text files).
>
> The main jobs.cfg file has been updated so that it also reads jobs from
> the ShaderCompilerTests directory.
> The file tasks.cfg has been extended with a task "rt_aqsl" that runs all
> shader compiler tests (you can use the same options for only running a
> subset of jobs as you can do with the other tasks).

I've set rt_aqsl to point to my development version, and run the tests.
It works very nicely, apart from the following observations:

* On my computer aqsl returns 255 for the error code - apparently the
  system is casting the expected return code of -1 to an unsigned char.
  I think we should change the return code to 1 for portability (I'll
  talk to Paul about this).
* The html reports "1 failures" on the index page, even when I have more
  than one failure.
* aqsl needs to be run with the -nocolor switch - I've changed this in
  the standard renderers.cfg.


I've also got a general request: It would be nice if the script printed
out a banner after a run to indicate the total number of failures, if
any.  If everything passed I don't have to look at the html at all.

> I have created two example jobs that demonstrate the usage. One test
> just compiles the plastic shader and checks its output whereas the other
> test checks aqsl's behavior in the case of a shader that contains an
> error (a missing semicolon).

Cool, these work here, except for the return code issue mentioned above.

> The job internally uses regular expressions to remove those parts of the
> output that differ from run to run (those parts are replaced by '*'
> characters).  In the case of stdout, it's the date stamp. On the
> compiled shaders, it's the Aqsis version.

Great, sounds very reasonable.  I like the idea of replacing the
changing stuff with * rather than just removing it.

> One somewhat weird behavior of aqsl is that in the case of an error
> inside the shader it just mentions the base shader name in the error
> message when the shader source begins within the first 2 lines of the
> source file. If the shader begins later, aqsl prints the full absolute
> path to the shader...?!?

Ugh, this is a bug.  I recall looking into this a while ago;
It's something to do with the file name being overwritten by a the full
path in some circumstances (I have a vague memory of it being to do with
the preprocessor).

At the time I concluded that it was tricky to fix for not much gain, but
now it seems like it should be fixed.


Again, thanks very much for implementing this, I think it will be quite
valuable :-)
~Chris.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Chris Foster]

On Mon, Aug 25, 2008 at 08:24:28PM +1000, Chris Foster wrote:
> I've set rt_aqsl to point to my development version, and run the tests.
> It works very nicely, apart from the following observations:
>
> * On my computer aqsl returns 255 for the error code - apparently the
>   system is casting the expected return code of -1 to an unsigned char.
>   I think we should change the return code to 1 for portability (I'll
>   talk to Paul about this).
> * The html reports "1 failures" on the index page, even when I have more
>   than one failure.

In fact, it seems like the index page is forgetting about all but the
first test case.  If I have failures in the second test case but not the
first, the index page reports "no failures".

~Chris.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Matthias Baas-3]

Chris Foster wrote:
> * The html reports "1 failures" on the index page, even when I have more
>   than one failure.

Oops, I was only counting shader mismatches, but not mismatches on the
return value, stdout or stderr. This is fixed now.

> * aqsl needs to be run with the -nocolor switch - I've changed this in
>   the standard renderers.cfg.

Actually, I was wondering that I didn't see any control characters on
Windows (so I didn't bother doing anything special about it). But of
course, using -nocolor makes sense.

> I've also got a general request: It would be nice if the script printed
> out a banner after a run to indicate the total number of failures, if
> any.  If everything passed I don't have to look at the html at all.

I have added a message that indicates whether there have been failures
or not. For the details, you still have to look at the html pages.

- Matthias -

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Chris Foster]

On Tue, Aug 26, 2008 at 10:55:06PM +0100, Matthias Baas wrote:
> > I've also got a general request: It would be nice if the script printed
> > out a banner after a run to indicate the total number of failures, if
> > any.  If everything passed I don't have to look at the html at all.
>
> I have added a message that indicates whether there have been failures
> or not. For the details, you still have to look at the html pages.

Thanks, this is nice.  Would it be easy to provide a summary of the
names of the failed test cases on the command line too?  I often have
one or two tests failing for known reasons.

~Chris.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Matthias Baas-3]

Chris Foster wrote:

> On Tue, Aug 26, 2008 at 10:55:06PM +0100, Matthias Baas wrote:
>>> I've also got a general request: It would be nice if the script printed
>>> out a banner after a run to indicate the total number of failures, if
>>> any.  If everything passed I don't have to look at the html at all.
>> I have added a message that indicates whether there have been failures
>> or not. For the details, you still have to look at the html pages.
>
> Thanks, this is nice.  Would it be easy to provide a summary of the
> names of the failed test cases on the command line too?  I often have
> one or two tests failing for known reasons.

I suppose it wouldn't be too difficult to add some sort of logging where
each task can write stuff into a log stream which is written to stdout
at the end.
I'm a bit hesitant though as I find this addition rather specific and
only useful in just the above case where you actually *expect* some
failures and the number of expected failures is fairly small. But in
that situation, you could instead just exclude the tests using the
-e/--exclude option.
Or would it work for you if you would just use lynx to dump the index
page? It doesn't contain the names of the failures, but you would see
the number of failures which basically gives you the same information.

Well, if all this is too inconvenient for some reason, I'm happy to add
that feature (I just want to make sure it's something that is really
needed and that it's also the best way to fix the problem).

- Matthias -

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Chris Foster]

On Tue, Sep 02, 2008 at 09:37:04PM +0100, Matthias Baas wrote:

> > Thanks, this is nice.  Would it be easy to provide a summary of the
> > names of the failed test cases on the command line too?  I often have
> > one or two tests failing for known reasons.
>
> I suppose it wouldn't be too difficult to add some sort of logging where
> each task can write stuff into a log stream which is written to stdout
> at the end.
> I'm a bit hesitant though as I find this addition rather specific and
> only useful in just the above case where you actually *expect* some
> failures and the number of expected failures is fairly small. But in
> that situation, you could instead just exclude the tests using the
> -e/--exclude option.

Hmm ok - I'd say that if it's going to be a kludge then don't bother.
I was imagining that it would be simple to walk the list of jobs *after*
they have been run, and just gather pass/fail information from there...
However I see that everything is rather geared toward html generation at
the moment.

Ideally what I'm after is the kind of output given by the ctest unit
testing utility distributed with cmake.  Typically it looks like the
following:

------------------------------------------------------------------------------
$ ctest
Start processing tests
Test project /home/foster/aqsis/branches/new_ribparser/build
  1/ 22 Testing aqsistypes/noise1234_test        Passed
  2/ 22 Testing aqsistypes/matrix_test           Passed
  3/ 22 Testing aqsistypes/color_test            Passed
  4/ 22 Testing aqsistypes/vector2d_test         Passed
  5/ 22 Testing aqsistypes/aqsismath_test        Passed
  6/ 22 Testing aqsistypes/vector4d_test         Passed
  7/ 22 Testing aqsistypes/cellnoise_test        Passed
  8/ 22 Testing aqsistypes/noise_test            Passed
  9/ 22 Testing aqsistypes/vector3d_test         Passed
 10/ 22 Testing aqsistypes/spline_test           Passed
 11/ 22 Testing aqsistypes/matrix2d_test         Passed
 12/ 22 Testing aqsistypes/enum_test             Passed
 13/ 22 Testing texlib/channellist_test          Passed
 14/ 22 Testing texlib/imagechannel_test         Passed
 15/ 22 Testing texlib/mixedimagebuffer_test     Passed
 16/ 22 Testing texlib/magicnumber_test          Passed
 17/ 22 Testing texlib/texfileheader_test        Passed
 18/ 22 Testing texlib/samplequad_test           Passed
 19/ 22 Testing texlib/tiffdirhandle_test     ***Failed
 20/ 22 Testing texlib/tiffinputfile_test        Passed
 21/ 22 Testing ribparse/riblexer_test           Passed
 22/ 22 Testing ribparse/ribparser_test          Passed

95% tests passed, 1 tests failed out of 22

The following tests FAILED:
         19 - texlib/tiffdirhandle_test (Failed)
Errors while running CTest

------------------------------------------------------------------------------

The point about this is the pass/fail status of each test is crystal
clear from the command line.

> Or would it work for you if you would just use lynx to dump the index
> page? It doesn't contain the names of the failures, but you would see
> the number of failures which basically gives you the same information.

TBH I can just as easily open index.html in firefox as dump it with
lynx.

> Well, if all this is too inconvenient for some reason, I'm happy to add
> that feature (I just want to make sure it's something that is really
> needed and that it's also the best way to fix the problem).

No, I definitely wouldn't say it's *really* needed.  I wouldn't spend
too much time on it (unless you want to ;-) ).

Thanks,
~Chris.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Matthias Baas-3]

Chris Foster wrote:

> On Tue, Sep 02, 2008 at 09:37:04PM +0100, Matthias Baas wrote:
>>> Thanks, this is nice.  Would it be easy to provide a summary of the
>>> names of the failed test cases on the command line too?  I often have
>>> one or two tests failing for known reasons.
>> I suppose it wouldn't be too difficult to add some sort of logging where
>> each task can write stuff into a log stream which is written to stdout
>> at the end.
>> I'm a bit hesitant though as I find this addition rather specific and
>> only useful in just the above case where you actually *expect* some
>> failures and the number of expected failures is fairly small. But in
>> that situation, you could instead just exclude the tests using the
>> -e/--exclude option.
>
> Hmm ok - I'd say that if it's going to be a kludge then don't bother.
> I was imagining that it would be simple to walk the list of jobs *after*
> they have been run, and just gather pass/fail information from there...

Well, that's the thing, it's not that easy. The jobs don't keep the
results themselves, it's all up to the task objects that operate on the
jobs.
The initial idea has been that there is a job list with jobs that can
produce RIB which can eventually be turned into one or more images. Then
  there are tasks which can be run on the jobs. As far as the main
program is concerned those tasks are pretty much black boxes that do
something with the jobs and eventually produce a html report. The
regression test task is just one task, but there are other tasks like
comparing images between renderers, comparing the performance between
renderers or between two versions of the same renderer or producing some
sort of RIB coverage statistics (which doesn't render an image at all).
That's basically why in the general case, the main program cannot
provide a summary of the test run, because it doesn't know at all what
those tasks were actually doing. And a job could have been rendered by
several renderers or even none at all. That's also why even the current
"OK"/"Failure" message at the end is already a bit "dirty" as this
message is specific to one single task and doesn't make sense for other
tasks that just don't have the concept of a "failure".

An additional obstacle is the recent introduction of the SL jobs that
don't really fit into the above scheme as they neither produce RIB nor
images. This is why they are completely separate from the rest of the
code. They are stored in a separate job list, are described by a
separate job class and can only be processed by the newly introduced
task class. The only thing those jobs share with the old jobs is the
mechanism to select them (the -i/-e options).

> Ideally what I'm after is the kind of output given by the ctest unit
> testing utility distributed with cmake.  Typically it looks like the
> following:
> [...]

I see what you mean. But the difference is that this unit test tool has
a smaller scope. It really is just a regression test tool, and it can
even work with text only.
Now I don't say that the rendertester tool really has to have that
broader scope, but this is just the way it is at the moment. And instead
of forcing one design (that even has flaws anyway) into a different
shape that it was never meant to be in, I would rather like to stop
here, review the requirements and missing features, drop features that
aren't used anyway and then rewrite it.

But I would rather only do that when it is really necessary and when it
would boost the workflow considerably (and so far, I was actually often
thinking about a rewrite, but the "critical mass" just hasn't been
reached yet. :)

>> Or would it work for you if you would just use lynx to dump the index
>> page? It doesn't contain the names of the failures, but you would see
>> the number of failures which basically gives you the same information.
>
> TBH I can just as easily open index.html in firefox as dump it with
> lynx.

Ah, I was thinking that maybe you are sometimes in a text only
environment or something like that... but as you are not, then yes, this
sounds like the most obvious solution. :)

- Matthias -

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Aqsis-development mailing list
Aqsis-development@...
https://lists.sourceforge.net/lists/listinfo/aqsis-development

[Chris Foster]

On Wed, Sep 03, 2008 at 10:59:17PM +0100, Matthias Baas wrote:

> Chris Foster wrote:
> > Hmm ok - I'd say that if it's going to be a kludge then don't bother.
> > I was imagining that it would be simple to walk the list of jobs *after*
> > they have been run, and