MWI Test Harness for CSS

[fantasai]

One way to get more people involved in CSS conformance testing
and to encourage implementors to send in implementation reports
would be creating a system that makes it easy for anyone to
submit pass/fail data. Lucky for us, the Mobile Web Initiative
has already created such a system. :)

There are some improvements I'd like to see before we start
using it to generate real implementation reports, however, the
first major one being to get the pass/fail buttons out of the
test file so they don't interfere with the test.

HP has volunteered to make improvements to the test harness,
and they asked me for a wishlist. Here's what I sent:

--------------------------------------------------------------

I promised you a test suite harness wishlist you could discuss.

But first, here are the links I sent over the telecon:

   CSS Test Suite wiki:
     http://csswg.inkedblade.net/test/css2.1
   Microsoft's tests:
     http://samples.msdn.microsoft.com/csstestpages/
   Mobile Web Initiative harness:
     http://www.w3.org/2007/03/mth/harness
   Mobile Web Initiative harness source code:
     http://dev.w3.org/cvsweb/2007/mobile-test-harness/

One of the very neat things about the MWI test harness is that it associates
the results with a user agent string. This means there's no need for users to
log in or to select their UA. They just load a test and click Pass/Fail/Can't
Tell.

Very briefly, the improvements I'd like to see are (numbered approximately
by priority):

New Harnesses:

   1. Harness using <iframe> to contain the test and pass/fail buttons on
      the containing page rather than inside the test file. (This format
      is good for desktop browsers.)

   2. Harness using links targetted at a new window to open the test and
      pass/fail buttons on the page containing the link rather than inside
      the test file. (This format is necessary for print.)

   In both cases the test itself should be referenced as a link, not fed through
   the CGI script. (This avoids tampering with the HTTP headers that normally
   get served up with the tests.)

   8. It would be nice if these harnesses could include some meta information
      about the test in addition to the buttons. E.g. the test ID (filename
      before extension), test title, any requirements documented in the test
      ("Warning: Must install Ahem font." etc). (I can extract this information
      for you into a flat-file database during the test suite build process
      so the harness doesn't have to do any analysis of the tests themselves.)

Better Reporting:

   3. Ability to consolidate results for various user agent strings under
      one category name. E.g. consolidate results for all UA strings that
      represent Opera 9.25 Beta 1 regardless of OS and localization.

   4. Pass/fail scores for the whole test suite

   5. Ability to report consolidated pass/fail scores for a named groups of
      tests. (To create e.g. a summary of what features are supported and to
      what level.)

   6. Interface for generating reports based on various parameters. URLs to
      these reports should be short and clean so they can be passed around
      in blogs/IM/email etc

   9. Prettier reports. :)

Data cleanup:

   7. Ability to delete all data for a given test (so that when a test is
      changed we can invalidate the results for that test). This process
      could be triggered manually on the command line rather than via CGI--
      that avoids the need for a login system.

~fantasai

[Berfanger, David]

Thanks to fantasai for opening this thread.

I am said volunteer from HP.

Since the initial wish list, one change and one addition to the wish list have been requested/discussed:

The change to the list is to use <object> rather than <iframe> to contain the test and pass/fail buttons;

The addition to the list is to offer editable text fields, so test results can be entered for tests that were conducted on user agents that differ from the one being used to enter the test data.

Also, the new test harness will need a database/php server similar to the Mobile Web Initiative.

Can Dominique, or anyone else, help with setting this up?

Finally, I'm taking suggestions on how best to collaborate on this development effort, in addition to posting to this public mailing list.

[fantasai]

Berfanger, David wrote:
> Thanks to fantasai for opening this thread.
>
> I am said volunteer from HP.
>
> Since the initial wish list, one change and one addition to the
> wish list have been requested/discussed:
>
> The change to the list is to use <object> rather than <iframe>
> to contain the test and pass/fail buttons;

To contain the test. The pass-fail buttons need to be outside the
<object> or they won't show up. :)

Tantek pointed to the CSS1 test suite pages as an example:
   http://www.w3.org/Style/CSS/Test/CSS1/20070302/sec15.htm
The <object> contains a link to the tests, so if the UA doesn't
support <object> the link is offered as a fallback.

We can replace the link navigation at the top with the pass/fail
buttons. (I can create a flat file database that will provide
the test title and any other metadata that's necessary: I don't
want the harness to depend on parsing the tests themselves.)

I've loaded the wishlist into the wiki:
   http://csswg.inkedblade.net/test/harness
Let me know if I missed anything. (Or you can edit it yourself. ;)

> The addition to the list is to offer editable text fields, so
> test results can be entered for tests that were conducted on user
> agents that differ from the one being used to enter the test data.
>
> Also, the new test harness will need a database/php server similar
> to the Mobile Web Initiative.
>
> Can Dominique, or anyone else, help with setting this up?
>
> Finally, I'm taking suggestions on how best to collaborate on this
> development effort, in addition to posting to this public mailing
> list.

Well, you can post sample output here for template critiques. For
the code itself, it would be good to get it into a publicly-accessible
VCS repository. Dominique can probably set you up with dev.w3.org CVS.

~fantasai

[Berfanger, David]

No arguments yet.

But, just wondering: Is there an official title for the test harness?

Something like "CSS Test Harness" rather than "Mobile Test Harness"?

[Dominique Hazael-Massieux-2]

Le lundi 21 avril 2008 à 12:17 -0700, fantasai a écrit :
> > Also, the new test harness will need a database/php server similar
> > to the Mobile Web Initiative.
> >
> > Can Dominique, or anyone else, help with setting this up?

Yes, I can help with that, although it would probably be easier if for
the time being (i.e. while you're experimenting/developing) you could
use a local database server.
 
> For
> the code itself, it would be good to get it into a publicly-accessible
> VCS repository. Dominique can probably set you up with dev.w3.org CVS.

David, can you send me an ssh2 key?

Dom

[Gabriele Romanato]

I don't understand why we're insisting on supporting non-standard elements such as <iframe>.
I don't understand why we're insisting on ignoring real world coverage tests (see Ian Hickson 'Testing methodology')
Finally, it's odd that a browser such as IE6 passes about 80% of the current CSS 2.1 test suite.
None of the tests provided up to now has been able to detect the effects of the hasLayout property in IE6.
What about importing all the test of www.brunildo.org/test ?
I think that this could be useful not only for browser vendors, but also for web developers and authors.
(Assuming that we're viewing the matter from a democratic point of view)



--
http://www.css-zibaldone.com/
http://www.css-zibaldone.com/test/ (English)
http://mimicry.css-zibaldone.com/

[Chris Wilson-12]

Some parts of this message have been removed. Learn more about Nabble's security policy.

Gabriele, I’m not sure why you’re calling <iframe> “non-standard”?  (http://www.w3.org/TR/html401/present/frames.html#h-16.5)

 

I don’t think the CSS Test Suite effort *IS* ignoring real world coverage tests – nor scenario-based testing.  There are a lot of features tests in the suite; I believe they need to be there.  That doesn’t replace scenario nor real-world tests; but it does seem logical to test the basics first.

 

Why would it be odd that IE6 passes 80% of the current CSS 2.1 test suite?  The suite is not complete.  As for the layout system in IE up through IE7 (of which hasLayout was a side effect), it was never intended to HAVE effects that would prevent it from passing any CSS test.

 

As for importing all of the tests from www.brunildo.org/test (or any other suite of tests out there), I think we all want the most comprehensive suite possible; all that is really needed is the author to enable an appropriate license on their tests, and someone to sign up to format them into the CSS 2.1 Test Suite (and the reviewers to review, and make sure they’re not just overlap with tests that are already there, of course).

 

-Chris

 

 

From: public-css-testsuite-request@... [mailto:public-css-testsuite-request@...] On Behalf Of Gabriele Romanato
Sent: Friday, April 25, 2008 12:04 PM
To: public-css-testsuite@...
Subject: Re: MWI Test Harness for CSS

 

I don't understand why we're insisting on supporting non-standard elements such as <iframe>.
I don't understand why we're insisting on ignoring real world coverage tests (see Ian Hickson 'Testing methodology')
Finally, it's odd that a browser such as IE6 passes about 80% of the current CSS 2.1 test suite.
None of the tests provided up to now has been able to detect the effects of the hasLayout property in IE6.
What about importing all the test of www.brunildo.org/test ?
I think that this could be useful not only for browser vendors, but also for web developers and authors.
(Assuming that we're viewing the matter from a democratic point of view)



--
http://www.css-zibaldone.com/
http://www.css-zibaldone.com/test/ (English)
http://mimicry.css-zibaldone.com/

[Alan Gresley-2]

Chris Wilson wrote:

> As for the layout system in IE up through IE7 (of which hasLayout was a side effect), it was never intended to HAVE effects that would prevent it from passing any CSS test.

Well since hasLayout is history there shouldn't be a need for testing
for hasLayout.


> As for importing all of the tests from www.brunildo.org/test<http://www.brunildo.org/test> (or any other suite of tests out there), I think we all want the most comprehensive suite possible; all that is really needed is the author to enable an appropriate license on their tests, and someone to sign up to format them into the CSS 2.1 Test Suite (and the reviewers to review, and make sure they're not just overlap with tests that are already there, of course).
>
> -Chris


To right there Chris. There Bruno Fassino's test cases.

http://www.brunildo.org/test/

My own.

http://css-class.com/test/

But none of these can simply be included in the CSS test suite. They are
intended for authors and not implementors. They are largely designed to
analyze or demonstrate why particular bug behavior happens (if that's
possible). I would like to have some of my test cases included but none
in their current format. Both Bruno and myself are very aware of what
duplications there are so it much simpler for us to breakdown these mass
test cases into individual test. Then there are the extra test cases of
bug behavior still in the pipeline.

Another issue is that some test cases are involving undefined behavior
in CSS2.1. This needs to be defined on the CSS WG list before such test
cases can be included in any CSS test suite.


Alan