Is test automation the holy grail?

[heusserm]

I just put up a new blog post I thought this group might enjoy:

http://xndev.blogspot.com/2008/10/complete-test-automation-is-holy-grail.html

But who cares what I think. What do -you- think?


--
Matthew Heusser,
Blog: http://xndev.blogspot.com

General, you are listening to a machine.  Do the world a favor and don't act
like one. -- Dr. Stephen Falken, "War Games", The Movie

[aidy lewis]

On 08/10/2008, Matt Heusser <matt.heusser@...> wrote:
>
> I just put up a new blog post I thought this group might enjoy:
>
>  http://xndev.blogspot.com/2008/10/complete-test-automation-is-holy-grail.html
>
>  But who cares what I think. What do -you- think?
>

Does this inlcude TDD or legacy automatic unit tests?

We have reasonable success with Watir\FireWatir and Rspec Cucumber,
even though at times painfully slow.

Aidy

[Eric Deslauriers]

Good one. The Holy Grail may also be a chalice or vessel and may not refer
to a physical object, but instead to a person.
I, personally, am a buff of all things Monty, so, perhaps it is only the
Grail-like beacon.

Eric D

On Wed, Oct 8, 2008 at 6:39 AM, Matt Heusser <matt.heusser@...> wrote:

>  I just put up a new blog post I thought this group might enjoy:
>
>
> http://xndev.blogspot.com/2008/10/complete-test-automation-is-holy-grail.html
>
> But who cares what I think. What do -you- think?
>
>
> --
> Matthew Heusser,
> Blog: http://xndev.blogspot.com
>
> General, you are listening to a machine.  Do the world a favor and don't
> act like one. -- Dr. Stephen Falken, "War Games", The Movie
>  
>

--
Eric D
08 K1200S Tricolor (phreowww)
06 Husqvarna TE610

[Eric Deslauriers]

Makes me think I should revive my C#  version of a Watir-like browser-driver
I'd written - I could run 20-30 instances of IE at the same time. With the
browsers hidden, I could still use it for other things. People would look at
me funny when my desktop would start making mad clicking sounds (I never got
around to figuring how to turn off the button-click sounds). I'd get a
worried look and say something like "Wow, that hard drive is really acting
up!!"
It was quite fast. I'd have to start over, though, since it's not my IP.

Any good resources for Watir support? It's a good productivity enhancer -
when I can get a new install to work.

I've started playing with it to automate menial tasks (Log into
Meetingplace, log into Teamtrack and bring up CR ID <x>, etc.) and it is
difficult at best to A) get working without some Ruby error and B) actually
recognize things on advanced pages.

For instance, I can not for the life of me figure out how to make it click
the <Attend> "button" in Meetingplace, instead I hacked it by sending <CR>.
Similar issues with Teamtrack, that one's not going well. I think I was
smarter when I was younger. Now I'm just wiser. :)

I've just about given up on it and am considering looking at other
command-line tools instead (if it's going to help me do my day job, it needs
to be unobtrusive...)

Eric D

On Wed, Oct 8, 2008 at 7:04 AM, aidy lewis <aidy.lewis@...>wrote:

> On 08/10/2008, Matt Heusser <matt.heusser@...> wrote:
> >
> > I just put up a new blog post I thought this group might enjoy:
> >
> >
> http://xndev.blogspot.com/2008/10/complete-test-automation-is-holy-grail.html
> >
> >  But who cares what I think. What do -you- think?
> >
>
> Does this inlcude TDD or legacy automatic unit tests?
>
> We have reasonable success with Watir\FireWatir and Rspec Cucumber,
> even though at times painfully slow.
>
> Aidy
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

--
Eric D
08 K1200S Tricolor (phreowww)
06 Husqvarna TE610

[aidy lewis]

Eric,

On 08/10/2008, Eric Deslauriers <eric.deslauriers@...> wrote:

> Makes me think I should revive my C#  version of a Watir-like browser-driver I'd written - I could run 20-30 instances of IE at the same time. With the browsers hidden, I could still use it for other things. People would look at me funny when my desktop would start making mad clicking sounds (I never got around to figuring how to turn off the button-click sounds). I'd get a worried look and say something like "Wow, that hard drive is really acting up!!"

Have you looked at Watin

http://watin.sourceforge.net/

Some projects here are using that, but I don't think they have got as
far as us using Ruby.

> Any good resources for Watir support? It's a good productivity enhancer - when I can get a new install to work.
>

Watir support is very active, on the mailing list and the Wiki

http://wiki.openqa.org/display/WTR/Project+Home

WatirCraft can provide professional support

http://www.watircraft.com/


> I've started playing with it to automate menial tasks (Log into Meetingplace, log into Teamtrack and bring up CR ID <x>, etc.) and it is difficult at best to A) get working without some Ruby error and B) actually recognize things on advanced pages.

Not really had this trouble. I use firebug to recognise html objects.
Regex internally built into the language is helpful to.
>
>
> For instance, I can not for the life of me figure out how to make it click the <Attend> "button" in Meetingplace, instead I hacked it by sending <CR>. Similar issues with Teamtrack, that one's not going well. I think I was smarter when I was younger. Now I'm just wiser. :)

The object could be nested and you may need to specify the full hierarchy.
>
>
> I've just about given up on it and am considering looking at other command-line tools instead (if it's going to help me do my day job, it needs to be unobtrusive...)
>

Aidy

[Bob Clancy]

I've added my comment to your blog.

-- Bob

On Wed, Oct 8, 2008 at  9:39 AM, Matt Heusser wrote:
I just put up a new blog post I thought this group might enjoy:
http://xndev. blogspot. com/2008/ 10/complete- test-automation-
is-holy-grail. html
<http://xndev.blogspot.com/2008/10/complete-test-automation-is-holy-grail.html>

[heusserm]

Eric asked:

>
>Does this inlcude TDD or legacy automatic unit tests?
>

Only if you are claiming to have /complete/ testing by such methods.

I am completely in favor of TDD and automated unit tests as
high-value, low-cost ways of improving velocity, enabling refactoring,
and increasingly quality.

It's when you say "and since I have these unit tests, I never had to
actually run this interactive application myself, at all, to push to
production" that I get suspicious.


--heusser
xndev.blogspotc.om

[Jeff Langr]

> It's when you say "and since I have these unit tests, I never had to
> actually run this interactive application myself, at all, to push to
> production" that I get suspicious.

Do people say that? I've not come across a shop that was so sure of what
they produced.

It should be obvious that unit tests alone, almost by definition, are
insufficient. I hope no one ever thinks they should ship software on the
basis of unit test execution itself. And I've never seen automated
acceptance tests, running against the deployed software, that could test
everything (that needed to be tested). Exploratory testing is always wise,
and there are just some aspects for which automation costs can outweigh
benefits ("look and feel" is potentially one of these).

Still, the Catholic refutation of the existence of the grail sounds to me
like typical VP speak, so I'm immediately suspect of their attempt to
sweep the legend under the carpet. :-)

The problem with grail seeking has generally been one of monomaniacal
obsession. Perhaps one of the best things that agile teams can do is avoid
false dichotomies, and instead seek to improve, simultaneously, their
firing quality on *all* cylinders.

The *search* for the real grail has close to zero benefit if it does not
exist. But even if there is no testing holy grail (where one click can
ship product), there is absolutely benefit in the search for it--the act
of improving and growing automation where it makes sense.

Jeff
http://langrsoft.com/agileJava

[Michael Bolton]

>Makes me think I should revive my C#  version of a Watir-like
browser-driver I'd written - I could run 20-30 instances of IE at the same
time. With the browsers hidden, I could still use it for other things.
People would look at me funny when my desktop would start making mad
clicking sounds (I never got around to figuring how to turn off the
button-click sounds).

Non-automated approach:  turn down the volume.

---Michael B.

[heusserm]

--- In agile-testing@..., "Jeff Langr" <jeff@...> wrote:
>
> And I've never seen automated acceptance tests, running
> against the deployed software, that could test everything
> (that needed to be tested).

Neither have I.  It doesn't stop people from claiming it's a
worthwhile goal.

>But even if there is no testing holy grail (where one click
>can ship product), there is absolutely benefit in the search
>for it--the act of improving and growing automation where
>it makes sense.

As long as you keep the "where it makes sense" part, I agree with you,
but then you're not searching for the Grail, you're doing something
that's actually a good idea.



--matt heusser

[Jeff Langr]

Greetings Matt & thanks for the reponse,

> --- In agile-testing@..., "Jeff Langr" <jeff@...> wrote:
>>But even if there is no testing holy grail (where one click
>>can ship product), there is absolutely benefit in the search
>>for it--the act of improving and growing automation where
>>it makes sense.
>
> As long as you keep the "where it makes sense" part, I agree with you,
> but then you're not searching for the Grail, you're doing something
> that's actually a good idea.

Good point, I thought about that after sending this message. Logic fault
on my part.

I'm unconvinced that a testing holy grail does not exist, particularly for
some applications. My problem with my own use of the phrase "where it
makes sense" is that it can make it too easy for some people to decide
they should just punt. I'd be much happier if people tried fairly hard
before giving up, because I think that's how we advance the craft. Someone
posted recently elsewhere that they had an application that didn't support
automated testing--why concede to that?

It's one of my problems with a number of things in software. "100%
coverage is impossible, so it's ok to choose what we want to unit test"
becomes an excuse that results in unnecessarily limited coverage using
test-after development (TAD). "You can't eliminate all comments" becomes
an excuse for people to not try and get rid of the comments that they well
could, were they to try a bit harder. And developers let methods grow to
quickly unreasonable sizes because they think it's impossible to have 1-6
line methods everywhere.

Jeff
http://langrsoft.com/agileJava

[C. Titus Brown]

On Wed, Oct 08, 2008 at 12:03:36PM -0500, Jeff Langr wrote:
-> > It's when you say "and since I have these unit tests, I never had to
-> > actually run this interactive application myself, at all, to push to
-> > production" that I get suspicious.
->
-> Do people say that? I've not come across a shop that was so sure of what
-> they produced.
->
-> It should be obvious that unit tests alone, almost by definition, are
-> insufficient. I hope no one ever thinks they should ship software on the
-> basis of unit test execution itself. And I've never seen automated
-> acceptance tests, running against the deployed software, that could test
-> everything (that needed to be tested). Exploratory testing is always wise,
-> and there are just some aspects for which automation costs can outweigh
-> benefits ("look and feel" is potentially one of these).

Is it appropriate to be Rumsfeldian here?  I sure hope so.

There's the stuff we know we know (addressed by unit and functional
tests), the stuff we know we don't know (addressed to some extent by
integration and smoke tests) and the stuff we don't know we don't know
(addressed by exploratory testing).

--titus
--
C. Titus Brown, ctb@...

[Steven Gordon-2]

On Wed, Oct 8, 2008 at 2:27 PM, C. Titus Brown <t@...> wrote:

> On Wed, Oct 08, 2008 at 12:03:36PM -0500, Jeff Langr wrote:
> -> > It's when you say "and since I have these unit tests, I never had to
>
> -> > actually run this interactive application myself, at all, to push to
> -> > production" that I get suspicious.
> ->
> -> Do people say that? I've not come across a shop that was so sure of what
> -> they produced.
> ->
> -> It should be obvious that unit tests alone, almost by definition, are
> -> insufficient. I hope no one ever thinks they should ship software on the
> -> basis of unit test execution itself. And I've never seen automated
> -> acceptance tests, running against the deployed software, that could test
> -> everything (that needed to be tested). Exploratory testing is always
> wise,
> -> and there are just some aspects for which automation costs can outweigh
> -> benefits ("look and feel" is potentially one of these).
>
> Is it appropriate to be Rumsfeldian here? I sure hope so.
>
> There's the stuff we know we know (addressed by unit and functional
> tests), the stuff we know we don't know (addressed to some extent by
> integration and smoke tests) and the stuff we don't know we don't know
> (addressed by exploratory testing).

I like this perspective (although it predates Rumsfeld).

When we discover stuff we did not know (through exploratory testing or
any other source of feedback), an important part of the repair process
should be to make sure that this particular stuff does not ever fool
us again by creating automated tests that fail due to what we
discovered.  We do that not only to verify that it has been fixed and
stays fixed, but also to document another thing that we now know the
system verifiable does do.

Steve

>
> --titus
> --
> C. Titus Brown, ctb@...

[Jeff Langr]

> On Wed, Oct 8, 2008 at 2:27 PM, C. Titus Brown <t@...> wrote:
>> Is it appropriate to be Rumsfeldian here? I sure hope so.
>>
>> There's the stuff we know we know (addressed by unit and functional
>> tests), the stuff we know we don't know (addressed to some extent by
>> integration and smoke tests) and the stuff we don't know we don't know
>> (addressed by exploratory testing).

I like the characterization too.

There's always a decision to accept some level of risk based on the
testing that could have been done but didn't get done. For some apps,
then, I believe the holy grail can exist: All automated tests pass (and
have been grown based off of learning from defects, per Steven's msg), all
new features have extremely low risk of breaking something, and the impact
of possible breakage is minimal. Therefore we are willing to convey that
risk to the customer.

Jeff

http://langrsoft.com/ourBooks.shtml#cleanCode

[Bob Clancy]

On Wed, Oct 8, 2008 at  6:21 PM, Jeff Langr wrote:
] There's always a decision to accept some level of risk based on the
] testing that could have been done but didn't get done. For some apps,
] then, I believe the holy grail can exist: All automated tests pass
(and
] have been grown based off of learning from defects, per Steven's msg),
all
] new features have extremely low risk of breaking something, and the
impact
] of possible breakage is minimal. Therefore we are willing to convey
that
] risk to the customer.

That's a lot of "if's".  I'd add that the "new features" need to be
isolated
from existing features or there's a danger of inintended side effects
you
might not be thinking about.  I'd also assert that if you have "new
features",
that the automated tests for the new features won't be perfect and catch
everything (no matter how hard you try) so manual exploratory testing is
still needed even in such contrived cases as this hypothetical one.

--
Bob

[Brian Marick]

On Oct 8, 2008, at 1:27 PM, Jeff Langr wrote:

> My problem with my own use of the phrase "where it
> makes sense" is that it can make it too easy for some people to decide
> they should just punt.

This is a really important point that people often miss. A lot of the  
Agile tradition, I confidently assert, goes like this:

1. There are statements X which are not true, but about which people  
say "wouldn't it be pretty to think so?" (Quote is from Ernest  
Hemingway, _The Sun Also Rises_) That is, look at a lot of the people  
who got a reputation as anti-automation zealots in the '90's. I was  
one of them. We didn't rule out automation from the get-go. Instead,  
we were convinced, by experience, that our hopes for automation were  
not realistic.

2. The Sensible Approach is to therefore accept the fact that X is not  
true and adapt to that world.

3. The Agile Approach is to change the world (or a local subset) to  
allow X to be true.

4. Once you've made X true, you can move forward in a more productive  
way. So, I claim, the early Agile obsessive insistence on automated  
testing has accomplished a lot, but has still conceded the need for  
manual exploratory testing. Now that we've pushed further, we can do a  
better job of folding exploratory testing into a project.

But I was always one for pushing a pendulum too far and letting  
reality pull you back to the center. The advantage of the Agile  
emphasis on short iterations and constant feedback is that it makes it  
safer to (temporarily) go to extremes.

-----
Brian Marick, independent consultant
Mostly on agile methods with a testing slant
www.exampler.com, www.exampler.com/blog, www.twitter.com/marick

[Michael Kernaghan-2]

--- In agile-testing@..., "Eric Deslauriers"
<eric.deslauriers@...> wrote:

"...I've started playing with it to automate menial tasks (Log into
Meetingplace, log into Teamtrack and bring up CR ID <x>, etc.) and it
is difficult at best to A) get working without some Ruby error and B)
actually recognize things on advanced pages.."

When I hit a Watir roadblock such as a modal dialog I cannot negotiate
I just complete the task manually (often just a click or two).

[Eric Deslauriers]

Suspicious? I get pretty antsy.
OTOH, no regression suite (manual or automated) and all you did was ad-hoc?
(don't laugh, it happens)

Eric

On Wed, Oct 8, 2008 at 8:44 AM, heusserm <matt.heusser@...> wrote:

> Eric asked:
>
> >
> >Does this inlcude TDD or legacy automatic unit tests?
> >
>
> Only if you are claiming to have /complete/ testing by such methods.
>
> I am completely in favor of TDD and automated unit tests as
> high-value, low-cost ways of improving velocity, enabling refactoring,
> and increasingly quality.
>
> It's when you say "and since I have these unit tests, I never had to
> actually run this interactive application myself, at all, to push to
> production" that I get suspicious.
>
>
> --heusser
> xndev.blogspotc.om
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

[Eric Deslauriers]

I always start with prioritized use cases (including misuse cases) for
testing, then decide how to implement the testing for them. Some get
automated, some are more cost effective over the long run to be manual over.
I'm a fan of ad-hoc (exploratory), but it's lower priority when you have a
good set of use cases and misuse cases; comprehensive structured testing
leveraging equivalence partitioning (requiring a good understanding of
product state related to test action) and boundary testing will gets me well
over the 80/20 usage model for the app.

For retail apps with a broad audience, it's hard to mimic your grandmother
or mother or dad or brother through structured interactions/tests, so
structured ad-hoc testing is a good thing to add in, but ROI can be low, so
invest wisely.

I'm not a fan of unstructured ad-hoc. Keeping track is cheap enough and
you'll need it anyways when you file your defects. Especially if it's pull
1, push 3, slide 6, push 1 devious to repro. :)

Eric D

On Wed, Oct 8, 2008 at 2:27 PM, C. Titus Brown <t@...> wrote:

> On Wed, Oct 08, 2008 at 12:03:36PM -0500, Jeff Langr wrote:
> -> > It's when you say "and since I have these unit tests, I never had to
> -> > actually run this interactive application myself, at all, to push to
> -> > production" that I get suspicious.
> ->
> -> Do people say that? I've not come across a shop that was so sure of what
> -> they produced.
> ->
> -> It should be obvious that unit tests alone, almost by definition, are
> -> insufficient. I hope no one ever thinks they should ship software on the
> -> basis of unit test execution itself. And I've never seen automated
> -> acceptance tests, running against the deployed software, that could test
> -> everything (that needed to be tested). Exploratory testing is always
> wise,
> -> and there are just some aspects for which automation costs can outweigh
> -> benefits ("look and feel" is potentially one of these).
>
> Is it appropriate to be Rumsfeldian here?  I sure hope so.
>
> There's the stuff we know we know (addressed by unit and functional
> tests), the stuff we know we don't know (addressed to some extent by
> integration and smoke tests) and the stuff we don't know we don't know
> (addressed by exploratory testing).
>
> --titus
> --
> C. Titus Brown, ctb@...
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

--
Eric D
08 K1200S Tricolor (phreowww)
06 Husqvarna TE610
<