Some thoughts on testings...

[Andrew Meit]

I recently had been forced to move all my files to my ibook and make  
it my main computer while I figure out what to do with my old iMac G4  
LCD dying...while doing so, I came across a file I saved and thought  
it important to post on the list in light of the open beta of  
2.7.5...read slowly:

Subject: Should Testers Be Allowed to Do Their Job?
From: ed@... (Edward V. Berard)
Date: Tue, 01 Aug 1995 07:35:23 -0400

Should testers be allowed to do their job? Before I can answer that,  
I need to define a few terms:

- To slightly paraphrase Glen Myers (author of "The Art of Software  
Testing"), testing is the process of examining some material with the  
intention (or goal) of finding errors. The "material" can include the  
material produced during the development part of the software life-
cycle (e.g., the products of the analysis, design, and coding  
efforts), and the specifications for the test cases themselves.

- Testing is a comparison process. Specifically, it compares one set  
of material (e.g., object code) with some correspondingly related  
material (e.g., the specification for that object code). Symptoms of  
errors are detected when what is expressed in one set of material  
does not agree with what is expressed in its corresponding material.  
(It is not the job of the tester to find the exact source of the  
error. That is the job of the creator (developer) of the material.)

- To be considered minimally adequate, the testing effort would have  
to detect the symptoms of all critical errors, and the symptoms of  
most, if not all, of the symptoms of all other errors.

I propose that the job of the tester is to do at least minimally  
adequate testing, as defined above.

Consider the following (unfortunately true) story. A few years ago, I  
was teaching a testing course at a firm that made implantable medical  
devices, e.g., pacemakers. Modern pacemakers not only contain  
software, but they are also programmed, monitored, and adjusted by  
systems that contain software. The people in the class were charged  
with testing the software associated with pacemaker systems.

- During the class, the students were required to specify test cases  
for software. Part of any test case is the purpose for that test  
case. One of the students insisted that "the purpose of test case xyz  
is to _prove_ _that_ _some_ _particular_ _function_ _is_  
_accomplished_." When I informed this person that, while this may  
have been the goal of the developer of the software, the goal of the  
tester was exactly the opposite.

- Later, in the same class, one of the testers noted that developers  
"felt bad" when they were informed of errors in the software products  
for which the developers were responsible. The tester said that she  
felt it was part of her job to provide some encouragement to the  
developers. Other testers admitted that they were often uncomfortable  
with being "the bearers of bad bad news" (i.e., reports of errors).

I was astonished. Was the ego of the developers more important than  
the health of the patients using the implanted medical devices? Were  
testers also required to be "psychological counselors" for the  
developers? Were both the testers and the developers aware of the  
negative impact on testing created by the reluctance of the testers  
to "make the developers feel bad?"

Should testers be allowed to do their job? Based on my experience  
over the past 15 years, the answer to that question in most shops is"  
"up to a point."

- Managers, developers, and testers often cannot distinguish between  
a _social_ situation and a _business_ situation. In a _social_  
context, I would agree that people must seriously consider whether it  
is worthwhile to point out "problems." Specifically, one must weigh  
the feelings of the individuals involved against any "improvements"  
that might result from bringing a problem to light. In a business  
situation the rules change. The parties to a business arrangement are  
supposed to be aware of the terms and conditions of the business  
arrangement, and are required to act accordingly.

One should _expect_ testers to find symptoms of errors. That is their  
job. Further, testers that routinely do not uncover symptoms of  
serious errors are not performing their jobs successfully.

When a tester informs a developer of the symptoms of the errors he or  
she has uncovered, the tester is _not_ making a statement about the  
basic goodness or badness of the developer as a human being. The  
tester _is_ making a statement about some material for which the  
developer is responsible.

- The time, staff, and other resources that are allocated to  
developers are often woefully inadequate. However, the time, staff,  
and resources that are allocated to the testing effort are, by in  
large, a joke. In all my years of training and consulting, I have run  
across fewer than 3 organizations that had anything even approaching  
an adequate assignment of resources to the testing effort.

It is almost as if the managers know this. When testers point out  
errors, that "slows down" the development effort. Inadequate testing  
resources result in fewer errors being reported, and, consequently,  
faster development.

- Many organizations make no secret of the fact that they consider  
developers to be more important than testers (or, for that matter,  
software quality assurance personnel, maintenance programmers, and  
others). I have had more than one manager inform me that "development  
requires true creativity, whereas testing does not require any  
special brilliance." The developers are often required to have  
university degrees, while many testers are non-degreed. Developers  
routinely receive more training, case tools, and other resources than  
do the testers. (At one of the public testing courses that I recently  
taught, one tester had a hard time believing that case tools for  
testers even existed.)

- There is also a prevailing attitude that a separate testing group  
is "a luxury." Many managers (and developers) reason that, in  
addition to creating the software product, developers can also  
adequately test that product. This line of reasoning is very wrong.  
Consider:

- Developers seldom have enough time to design and implement the  
product, much less to test the product. (In fact, one of the most  
common reasons for the creation of a separate testing group is "to  
relieve the pressure on the developers.")

- Developers are seldom trained in software testing techniques. This  
often results in largely unfocused, hit or miss testing efforts.  
(When developers attend a software testing course, they are often  
astonished by the amount of work required for even minimal testing.)

- Of course, developers suffer from all the problems of human beings  
who are asked to evaluate their own efforts, e.g., lack of  
objectivity, psychological blindness (to their own errors), and lack  
of perspective.

Given the attitude that a separate testing group is a luxury, it is  
quite normal to deny resources to this group, or to cut it out  
entirely when budget constraints loom large.

It is very sad to see a group of people who are given an assignment,  
and are then prevented from carrying out that assignment to the  
fullest of their abilities.

-- Ed

---------------------------------------------------------------------
Edward V. Berard | Voice: [phone] The Object Agency, Inc. |  
Fax: [phone] Lakeforest Blvd., Suite 380 | E-Mail:  
ed@... Gaithersburg, Maryland 20877

Shalom, Andrew
{Choose Life, Create Hope, Nurture Love...}



_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[J. Landman Gay]

Andrew Meit wrote:
> I recently had been forced to move all my files to my ibook and make it
> my main computer while I figure out what to do with my old iMac G4 LCD
> dying...while doing so, I came across a file I saved and thought it
> important to post on the list in light of the open beta of 2.7.5...read
> slowly:

This post was just excellent, thanks for passing it on. I couldn't agree
more with the points the author makes. In almost every professional job
I've done, the client says they will test the software themselves. This
is usually disasterous. No one seems to recognize that the testing
process should take as long as the development process and should be an
integral part of the budget and schedule. Few realize the kind of talent
and precision that goes into professional software testing.

I am lucky enough to have a wonderful professional QA person who works
with me when the client's budget allows it. I understand the author's
comments about "hurting the developer's feelings" -- I always knew when
my tester was at work because my mailbox would fill up with bug reports.
My response was always a love/hate thing. I loved that he was so good at
what he does and that he was finding flaws, because in the long run it
would make my work much better. But I also hated that he was finding
flaws because it meant more work for me. Every time I'd see another 20
reports in my mailbox my heart would sink. And then I'd have to go fix
things again.

But he's a great guy, and we're both pros, and we are able to easily
separate critique from criticism. As a result, my last project was one
of the best of my career. Note that this client originally told me he'd
test it himself. He found three or four bugs over a couple of weeks. I
insisted that we needed a professional QA engineer. Once my QA guy got
his hands on it, he found 165 bugs.

The client was astounded -- and grateful. There is a skill and talent
involved in software testing, and not everyone has what it takes to do
it well. The developer certainly doesn't, and the client rarely does
either. A qualified, professional QA person is pure gold, and if you
find a good one you should do whatever it takes to hang on to them.

--
Jacqueline Landman Gay         |     jacque@...
HyperActive Software           |     http://www.hyperactivesw.com
_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Rob Cozens]

Andrew, Jacque, et al:

>- To be considered minimally adequate, the testing effort would have
>to detect the symptoms of all critical errors, and the symptoms of
>most, if not all, of the symptoms of all other errors.
>
>I propose that the job of the tester is to do at least minimally
>adequate testing, as defined above.

Would anyone care to comment on the potential role of testers to:

A.  Point out where the operational manifestation of the
specifications creates illogical or inappropriate user experiences and

B.  Suggest alternative designs or approaches?

Rob Cozens
CCW, Serendipity Software Company

"And I, which was two fooles, do so grow three;
Who are a little wise, the best fooles bee."

from "The Triple Foole" by John Donne (1572-1631)

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Mark Wieder]

Rob-

Sunday, January 14, 2007, 8:37:25 AM, you wrote:

> Would anyone care to comment on the potential role of testers to:

> A.  Point out where the operational manifestation of the
> specifications creates illogical or inappropriate user experiences and

I think this is key to the user experience. If a user gets confused by
the UI or finds an ambiguous situation then the app is not designed
properly. Ideally you want this to show up in testing, before the app
goes out the door. A good tester will flag situations in which, for
example, modality hides or makes unavailable needed functions; label
text is confusing; or it's not clear what the next step should be.
This is one of the exceptions to part B below.

> B.  Suggest alternative designs or approaches?

Nope. This shouldn't be a function of testing. The problem here is
that if the tester/user gets involved in the design of the app they
are then in the same myopic position as the developer. Then you have
to get someone else to test it. You always know your own code much too
well to be able to test it thoroughly.

There are exceptions, of course. I've had to go back to developers and
have them expose a particular functionality or display a value in a
text field instead of a label field so that I could validate a change
of state or a data point.

--
-Mark Wieder
 mwieder@...

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[Mark Wieder]

Andrew-

Saturday, January 13, 2007, 9:35:09 AM, you wrote:

> It is almost as if the managers know this. When testers point out
> errors, that "slows down" the development effort. Inadequate testing
> resources result in fewer errors being reported, and, consequently,
> faster development.

ROTFL. One classic piece of literature addressing this point is H.
James Harrington's "Poor-Quality Cost".

--
-Mark Wieder
 mwieder@...

_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution

[david bovill]

Just to add to this thread - a little tangentially maybe - a lot further can
be got within the Rev development community by using the community to
develop robust components and libraries. By this I mean developer testing
using something akin to the philosophy of open source development and unit
tests.

This would not replace user testing in the sense being discussed - it would
just pick up some of the bugs before this stage. The main advantage of this
would be the cost - the testing comes as a by=product of developers using
code. This is not happening a the moment while it does in other coding
environments - which is why they have extensive libraries of well tested
code. Of course well tested depends on both the community concerned and the
testing and developing infrastructure they use?
_______________________________________________
use-revolution mailing list
use-revolution@...
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution