Testing SHOULD

[fantasai]

There are three levels of requirement in the CSS specs

MUST - the behavior is required
SHOULD/RECOMMENDED - the behavior is required unless there's a
                      good reason not to do it
MAY - the behavior is allowed

We're testing as many MUST requirements as we can, of course,
but it would be useful to also test SHOULD requirements. However,
we should distinguish these tests somehow from the MUST tests.
I was thinking to have a flag for SHOULD tests. Not sure what
to call it... optional? should? recommend?

Comments?

~fantasai

[Alan Gresley-2]

fantasai wrote:

>
> There are three levels of requirement in the CSS specs
>
> MUST - the behavior is required
> SHOULD/RECOMMENDED - the behavior is required unless there's a
>                      good reason not to do it
> MAY - the behavior is allowed
>
> We're testing as many MUST requirements as we can, of course,
> but it would be useful to also test SHOULD requirements. However,
> we should distinguish these tests somehow from the MUST tests.
> I was thinking to have a flag for SHOULD tests. Not sure what
> to call it... optional? should? recommend?
>
> Comments?
>
> ~fantasai

I would say RECOMMENDED since 'should' or similar 'could' are quite weak
words considering there 'should' be good reason not to do it (the
behavior). For MAY I 'would' like OPTIONAL instead.


Alan

[Bonner, Matt (IPG)]

Thinking about how the SHOULD tests will help might offer
guidance for how to name them.

Obviously, these tests serve to see how UA's are doing; the
question that seems interesting to consider is what people
will conclude based on the results.

For example, if the results will guide decisions on which
SHOULD items should get promoted to MUST or dropped to MAY,
a flag name suggesting that idea seems helpful.

How do people expect testing SHOULD requirements to help?

Matt
--
Matt Bonner
Hewlett-Packard Company

-----Original Message-----
From: public-css-testsuite-request@... [mailto:public-css-testsuite-request@...] On Behalf Of fantasai
Sent: Friday, May 02, 2008 2:29 PM
To: public-css-testsuite@...
Subject: Testing SHOULD


There are three levels of requirement in the CSS specs

MUST - the behavior is required
SHOULD/RECOMMENDED - the behavior is required unless there's a
                      good reason not to do it
MAY - the behavior is allowed

We're testing as many MUST requirements as we can, of course,
but it would be useful to also test SHOULD requirements. However,
we should distinguish these tests somehow from the MUST tests.
I was thinking to have a flag for SHOULD tests. Not sure what
to call it... optional? should? recommend?

Comments?

~fantasai

[L. David Baron]

On Saturday 2008-05-03 10:56 +1000, Alan Gresley wrote:
> fantasai wrote:
>>
>> There are three levels of requirement in the CSS specs
>>
>> MUST - the behavior is required
>> SHOULD/RECOMMENDED - the behavior is required unless there's a
>>                      good reason not to do it
>> MAY - the behavior is allowed

> I would say RECOMMENDED since 'should' or similar 'could' are quite weak
> words considering there 'should' be good reason not to do it (the
> behavior). For MAY I 'would' like OPTIONAL instead.

These terms aren't up for debate.  They've been standardized by RFC
2119 for over ten years: http://www.ietf.org/rfc/rfc2119.txt .  Nor
is a spec-wide editorial rewrite (to prefer some RFC 2119 terms over
others) of CSS 2.1 appropriate at its current maturity level (nor is
this the appropriate list to raise such an issue).

Using "SHOULD" and "MAY" is often much more concise than "OPTIONAL"
and "RECOMMENDED", and experienced spec readers ought to know what
they mean (or notice the text at the beginning pointing to RFC
2119).

-David

--
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/

[Alan Gresley-2]

>> I would say RECOMMENDED since 'should' or similar 'could' are quite weak
>> words considering there 'should' be good reason not to do it (the
>> behavior). For MAY I 'would' like OPTIONAL instead.
>
> These terms aren't up for debate.  They've been standardized by RFC
> 2119 for over ten years: http://www.ietf.org/rfc/rfc2119.txt.  Nor
> is a spec-wide editorial rewrite (to prefer some RFC 2119 terms over
> others) of CSS 2.1 appropriate at its current maturity level (nor is
> this the appropriate list to raise such an issue).
>
> Using "SHOULD" and "MAY" is often much more concise than "OPTIONAL"
> and "RECOMMENDED", and experienced spec readers ought to know what
> they mean (or notice the text at the beginning pointing to RFC
> 2119).
>
> -David

Well pardon my ignorance about a standardization of terms dating from
1997. I am only commenting from what I believe to be best personally. I
am also commenting from the perspective of what these terms mean in
Australian English. SHOULD is a weak word and RECOMMENDED is a strong
word when spoken by the average Australian.

All I did was provided my comments from my perspective, life experience
and knowledge. Do Americans set the standards for the rest of the world
which every other person must adhere to on this planet?


Alan

[L. David Baron]

On Saturday 2008-05-03 13:22 +1000, Alan Gresley wrote:
> Well pardon my ignorance about a standardization of terms dating from 1997.
> I am only commenting from what I believe to be best personally. I am also
> commenting from the perspective of what these terms mean in Australian
> English. SHOULD is a weak word and RECOMMENDED is a strong word when spoken
> by the average Australian.
>
> All I did was provided my comments from my perspective, life experience and
> knowledge. Do Americans set the standards for the rest of the world which
> every other person must adhere to on this planet?

They don't match American English perfectly either.  (And some, like
"MUST NOT", are particularly confusing for speakers of some other
languages.)  However, they're a widely-recognized standard for how
to write standards, and this is definitely not the correct forum for
discussions of changing them.

-David

--
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/

[fantasai]

L. David Baron wrote:

> On Saturday 2008-05-03 10:56 +1000, Alan Gresley wrote:
>> fantasai wrote:
>>> There are three levels of requirement in the CSS specs
>>>
>>> MUST - the behavior is required
>>> SHOULD/RECOMMENDED - the behavior is required unless there's a
>>>                      good reason not to do it
>>> MAY - the behavior is allowed
>
>> I would say RECOMMENDED since 'should' or similar 'could' are quite weak
>> words considering there 'should' be good reason not to do it (the
>> behavior). For MAY I 'would' like OPTIONAL instead.
>
> These terms aren't up for debate.  They've been standardized by RFC
> 2119 for over ten years: http://www.ietf.org/rfc/rfc2119.txt .  Nor
> is a spec-wide editorial rewrite (to prefer some RFC 2119 terms over
> others) of CSS 2.1 appropriate at its current maturity level (nor is
> this the appropriate list to raise such an issue).
>
> Using "SHOULD" and "MAY" is often much more concise than "OPTIONAL"
> and "RECOMMENDED", and experienced spec readers ought to know what
> they mean (or notice the text at the beginning pointing to RFC
> 2119).

Er.. a) we're talking about flags for tests, not wording for specs and
b) Alan's suggestion matches RFC2119 anyway.

~fantasai

[fantasai]

Bonner, Matt (IPG) wrote:

> Thinking about how the SHOULD tests will help might offer
> guidance for how to name them.
>
> Obviously, these tests serve to see how UA's are doing; the
> question that seems interesting to consider is what people
> will conclude based on the results.
>
> For example, if the results will guide decisions on which
> SHOULD items should get promoted to MUST or dropped to MAY,
> a flag name suggesting that idea seems helpful.
>
> How do people expect testing SHOULD requirements to help?

I think the main goal is to drive interoperability on implementing
the SHOULD requirements. Dropping SHOULD requirements based on
non-implementation is also a possibility, but something we want
to avoid. :)

~fantasai

[fantasai]

Alan Gresley wrote:

>
> fantasai wrote:
>>
>> There are three levels of requirement in the CSS specs
>>
>> MUST - the behavior is required
>> SHOULD/RECOMMENDED - the behavior is required unless there's a
>>                      good reason not to do it
>> MAY - the behavior is allowed
>>
>> We're testing as many MUST requirements as we can, of course,
>> but it would be useful to also test SHOULD requirements. However,
>> we should distinguish these tests somehow from the MUST tests.
>> I was thinking to have a flag for SHOULD tests. Not sure what
>> to call it... optional? should? recommend?
>>
>> Comments?
>
> I would say RECOMMENDED since 'should' or similar 'could' are quite weak
> words considering there 'should' be good reason not to do it (the
> behavior). For MAY I 'would' like OPTIONAL instead.

Ok, I'm adding two flags to the list:

rec
   Behavior tested is RECOMMENDED, but not REQUIRED. [RFC2119]
optional
   Behavior tested is preferred but OPTIONAL. [RFC2119] (These tests
   must be reviewed by a test suite owner or peer.)

I shortened 'recommended' to 'rec' in the interests of
   a) keeping it short
   b) avoiding spelling mistakes :)

'Optional' should be added only if the behavior tested is the preferred
behavior: we don't want to encourage implementors to pass tests on
discouraged behavior. (If there's a case where neither behavior is preferred,
then both behaviors should be tested.) These tests require test suite
Owner/Peer review because which behavior is preferred is not always
obvious.

Does that sound reasonable? (Arron?)

~fantasai

[L. David Baron]

On Saturday 2008-05-03 08:50 -0700, fantasai wrote:
> Ok, I'm adding two flags to the list:
>
> rec
>   Behavior tested is RECOMMENDED, but not REQUIRED. [RFC2119]
> optional
>   Behavior tested is preferred but OPTIONAL. [RFC2119] (These tests
>   must be reviewed by a test suite owner or peer.)

rec could easily be confused with the W3C REC (Recommendation)
status.

I prefer should and may given that the spec-writing community tends
to refer to these requirement levels as shoulds and mays.

-David

--
L. David Baron                                 http://dbaron.org/
Mozilla Corporation                       http://www.mozilla.com/

[fantasai]

L. David Baron wrote:

> On Saturday 2008-05-03 08:50 -0700, fantasai wrote:
>> Ok, I'm adding two flags to the list:
>>
>> rec
>>   Behavior tested is RECOMMENDED, but not REQUIRED. [RFC2119]
>> optional
>>   Behavior tested is preferred but OPTIONAL. [RFC2119] (These tests
>>   must be reviewed by a test suite owner or peer.)
>
> rec could easily be confused with the W3C REC (Recommendation)
> status.
>
> I prefer should and may given that the spec-writing community tends
> to refer to these requirement levels as shoulds and mays.

Ok, changed.

~fantasai

[Arron Eicholz]

fantasai wrote:

> Ok, I'm adding two flags to the list:
>
> rec
>    Behavior tested is RECOMMENDED, but not REQUIRED. [RFC2119]
> optional
>    Behavior tested is preferred but OPTIONAL. [RFC2119] (These tests
>    must be reviewed by a test suite owner or peer.)
>
> I shortened 'recommended' to 'rec' in the interests of
>    a) keeping it short
>    b) avoiding spelling mistakes :)
>
> 'Optional' should be added only if the behavior tested is the preferred
> behavior: we don't want to encourage implementors to pass tests on
> discouraged behavior. (If there's a case where neither behavior is
> preferred,
> then both behaviors should be tested.) These tests require test suite
> Owner/Peer review because which behavior is preferred is not always
> obvious.
>
> Does that sound reasonable? (Arron?)

Yes this makes sense to me. Just to clarify though do SHOULD rules fall into the OPTIONAL category?

There are 10 keywords that are 'recommended' for use when describing information in the spec do we plan on adding flags for any additional keywords?

What is the exact mapping?

Keyword         Flag
MUST
MUST NOT
REQUIRED
SHALL
SHALL NOT
SHOULD
SHOULD NOT
RECOMMENDED             rec
MAY
OPTIONAL                optional

I think the first five (MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT) do not need flags. The remaining ones I think do but I am unsure if we want to map all of the remaining ones to the flag 'optional'. Technically all of the remaining five keywords are optional for user agents. There is nothing saying they have to follow the recommended solution though it is probably in their best interest to do so.

So I am left with this...

Personally I think SHOULD, SHOULD NOT, and RECOMMENDED are closely related and should all map to the same flag. I do not think 'rec' is the right flag for this since it can easily be confused to mean a W3C REC (Recommendation) status just like David stated. I would suggest the flag 'preferred'. The only problem I see with that flag is people will misspell it but that is true with almost any flag.

The MAY and OPTIONAL keywords, I think the flag 'optional' is fine for these. I don't see a need to separate those into separate flags.

--
Thanks,
Arron Eicholz

[fantasai]

Arron Eicholz wrote:

> fantasai wrote:
>> Ok, I'm adding two flags to the list:
>>
>> rec
>>    Behavior tested is RECOMMENDED, but not REQUIRED. [RFC2119]
>> optional
>>    Behavior tested is preferred but OPTIONAL. [RFC2119] (These tests
>>    must be reviewed by a test suite owner or peer.)
>>
>> I shortened 'recommended' to 'rec' in the interests of
>>    a) keeping it short
>>    b) avoiding spelling mistakes :)
>>
>> 'Optional' should be added only if the behavior tested is the preferred
>> behavior: we don't want to encourage implementors to pass tests on
>> discouraged behavior. (If there's a case where neither behavior is
>> preferred,
>> then both behaviors should be tested.) These tests require test suite
>> Owner/Peer review because which behavior is preferred is not always
>> obvious.
>>
>> Does that sound reasonable? (Arron?)
>
> Yes this makes sense to me. Just to clarify though do SHOULD rules fall
> into the OPTIONAL category?

No.

> There are 10 keywords that are 'recommended' for use when describing
> information in the spec do we plan on adding flags for any additional
> keywords?

No.

> What is the exact mapping?

Defined by RFC2119.
   http://www.ietf.org/rfc/rfc2119.txt

> I think the first five (MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT) do
> not need flags.

Agreed.

> The remaining ones I think do but I am unsure if we want to map all of
> the remaining ones to the flag 'optional'. Technically all of the
> remaining five keywords are optional for user agents. There is nothing
> saying they have to follow the recommended solution though it is probably
> in their best interest to do so.
>
> So I am left with this...
>
> Personally I think SHOULD, SHOULD NOT, and RECOMMENDED are closely
> related and should all map to the same flag. I do not think 'rec' is
> the right flag for this since it can easily be confused to mean a W3C
> REC (Recommendation) status just like David stated. I would suggest
> the flag 'preferred'. The only problem I see with that flag is people
> will misspell it but that is true with almost any flag.
>
> The MAY and OPTIONAL keywords, I think the flag 'optional' is fine for
> these. I don't see a need to separate those into separate flags.

The mapping is given in RFC2119

MUST == SHALL == REQUIRED
SHOULD == RECOMMENDED
MAY == OPTIONAL

As David suggested, I added the flags 'should' and 'may' to the list.
The default (no flag) means it's required.

~fantasai