Test case modularity

[Alexandre Ferrieux]

With the [chan pipe] patch and also with other, not-yet-TIPped stuff
of mine, I noticed that the checking of error messages in the test
suite is often made with strict string equality. This makes sense for
many messages which are likely frozen for eternity, but there is a
specific subpopulation which is more dynamic: messages enumerating
subcommands of an ensemble.

This means that every extension of an ensemble in the core needs
fixing test cases that are completely unrelated to the new subcommand:
tests expecting the message

    bad subcommand "foo": must be a, b, c, or d.

must be updated to check for

    bad subcommand "foo": must be a, b, c, d, or e.

Is there a good reason to do so ? Why not [regexp {^bad subcommand}] ?

-Alex

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08
_______________________________________________
Tcl-Core mailing list
Tcl-Core@...
https://lists.sourceforge.net/lists/listinfo/tcl-core

[Donal K. Fellows-2]

Alexandre Ferrieux wrote:

> This means that every extension of an ensemble in the core needs
> fixing test cases that are completely unrelated to the new subcommand:
> tests expecting the message
>
>     bad subcommand "foo": must be a, b, c, or d.
>
> must be updated to check for
>
>     bad subcommand "foo": must be a, b, c, d, or e.
>
> Is there a good reason to do so ? Why not [regexp {^bad subcommand}] ?

I've wondered about that in the past too. I'm not sure; perhaps it is
wiser to allow the checking that the list of subcommands is what it is
expected to be as it helps enforce coverage. But if you do want to do a
non-exact match, please do this instead:

   -match glob -result {bad subcommand "foo": must be *}

It's easier to read, and doesn't obscure what the string was when a
failure happens. A more complete example might be this:

   test chan-0.1 {bad chan subcommand} -returnCodes error -body {
       chan foobar
   } -match glob -result {bad subcommand "foobar": must be *}

Let's compare that with the traditional way of doing this (assuming
we're doing glob matching; regexps don't make this easier).

   test chan-0.1 {bad chan subcommand} {
       string match {1 {bad subcommand "foobar": must be *}} \
               [list [catch {chan foobar} msg] $msg]
   } 1

I think the new-style one is clearer, especially when things go wrong!
YMMV (but I hope not...)

Donal.

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08
_______________________________________________
Tcl-Core mailing list
Tcl-Core@...
https://lists.sourceforge.net/lists/listinfo/tcl-core

[dgp-4]

Alexandre Ferrieux wrote:

> This means that every extension of an ensemble in the core needs
> fixing test cases that are completely unrelated to the new subcommand:
> tests expecting the message
>
>     bad subcommand "foo": must be a, b, c, or d.
>
> must be updated to check for
>
>     bad subcommand "foo": must be a, b, c, d, or e.
>
> Is there a good reason to do so ? Why not [regexp {^bad subcommand}] ?

Let's take a step back away from implementation details to principles.

A test ought to exist for a purpose.  It ought to be testing something.

A test ought to be associated with some code getting tested.  The Tcl
test suite tends to follow the convention that tests in a file named
something like foo.test are for testing the code in tclFoo.c.

So if your test passes or fails based on the exact contents of an error
message, you are testing the code that constructs that error message,
whether that's your intent or not.

Thus, the text of an error message ought to be tested *at most* by
tests associated with the source code file where the content
of the error message is determined.  It makes no sense to test the
contents or formatting of an error message anywhere else.  All it does
is make the tests fragile against changes in code that in principle is
not being tested.

It may not be worthwhile to test error message contents at all, but that's
a judgment for the maintainer of the tclFoo.c file to make.

Within that limited context, yes, it makes sense to use regexp or
glob matching to check the basic formatting of the error message without
checking the entire contents in a way that requires maintenance.  It
may make sense for a very limited number of tests to check the whole
subcommand list (along with the associated maintenance) if the maintainer
has reason to guard against errors or inconsistencies in that part of
the code.  There's no need for more tests than cover the cases though.

--
| Don Porter          Mathematical and Computational Sciences Division |
| donald.porter@...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08
_______________________________________________
Tcl-Core mailing list
Tcl-Core@...
https://lists.sourceforge.net/lists/listinfo/tcl-core

[Alexandre Ferrieux]

On 7/5/08, Donal K. Fellows <donal.k.fellows@...> wrote:
>
>  test chan-0.1 {bad chan subcommand} -returnCodes error -body {
>      chan foobar
>  } -match glob -result {bad subcommand "foobar": must be *}
>

FWIW, I have updated the ref impl of 304 to include these more generic
(and more modern) test cases in the three places where the addition of
[chan pipe] required updating anyway.

Also, thanks to TCT members who have already voted for 304.
I will be offline from now to July 21st, so when the deadline comes on
next Wednesday, if the outcome is positive, feel free to commit for
me.

-Alex

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08
_______________________________________________
Tcl-Core mailing list
Tcl-Core@...
https://lists.sourceforge.net/lists/listinfo/tcl-core