Need your help in evaluating TestNG.

[kiri-3]

I was evaluating TestNG to use it in one of my projects to achieve
functional testing and stress testing. What I discovered are the
following things.

1. We treat the whole Class as a functional test which includes
setup(), test() cleanup() and some helper methods so there will be
only one method in super class ‘executeTest()’ which needs to be
called, And the Test Class many be at 4 level down the hierarchy as
tests needs complex code which are modularized. So it is not possible
to tag individual tests with different groups (we need to override
executeTest() in each Class to do this) as we are going to write
around 1000 tests it’s not manageable if the test groups needs to be
modified frequently for many number of tests.

2. Lets say I have to tag the tests with
Product versions (PV01, PV02, PV03, PV04)
Applicable clients (C1,C2,C3,C4) and many more

So I want a way to configure the mapping between Test Class and the
applicable tags in single place
Ex: com.module.submodule.Pos001 = PV01,PV02,C1,C4,
com.module.submodule.Pos002 = PV01,C1,C2,C3. in a property file, and
generate the xmlSuite for suites (PV01 && C1) , (PV02 && C2) etc and
run the generated suites through TestNg. So that I have fine grain
control on what I am going to run. Here I am not tagging the tests but
need to make sure that the test belongs to groups as specified in
property file dynamically. Now let’s say 20 tests are also applicable
for C5, I just need to modify this file. Is there a way to do it?

3. There will be many tests which will be applicable for many
combination of product and clients and the Exceptions depends on the
product on which we run the test, so @Test(expectedExceptions = "...")
is of less use for negative testing.
4. Also we want to keep the test log in a separate file for each test
so that we can refer to it at a later point of time and should have an
hyperlink to this file from the generated report.

So Please let me know is it feasible to use TestNG for above
requirements?

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@...
To unsubscribe from this group, send email to testng-users+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[kiri-3]

Expecting a comment at least :-)

On Nov 10, 1:53 pm, kiri <kiran.eshwara...@...> wrote:

> I was evaluating TestNG to use it in one of my projects to achieve
> functional testing and stress testing. What I discovered are the
> following things.
>
> 1.      We treat the whole Class as a functional test which includes
> setup(), test() cleanup() and some helper methods so there will be
> only one method in super class ‘executeTest()’ which needs to be
> called, And the Test Class many be at 4 level down the hierarchy as
> tests needs complex code which are modularized. So it is not possible
> to tag individual tests with different groups (we need to override
> executeTest() in each Class to do this) as we are going to write
> around 1000 tests it’s not manageable if the test groups needs to be
> modified frequently for many number of tests.
>
> 2.      Lets say I have to tag the tests with
> Product versions (PV01, PV02, PV03, PV04)
> Applicable clients (C1,C2,C3,C4) and many more
>
> So I want a way to configure the mapping between Test Class and the
> applicable tags in single place
> Ex: com.module.submodule.Pos001 = PV01,PV02,C1,C4,
> com.module.submodule.Pos002 = PV01,C1,C2,C3. in a property file, and
> generate the xmlSuite for suites (PV01 && C1) , (PV02 && C2) etc and
> run the generated suites through TestNg. So that I have fine grain
> control on what I am going to run. Here I am not tagging the tests but
> need to make sure that the test belongs to groups as specified in
> property file dynamically. Now let’s say 20 tests are also applicable
> for C5, I just need to modify this file. Is there a way to do it?
>
> 3.      There will be many tests which will be applicable for many
> combination of product and clients and the Exceptions depends on the
> product on which we run the test, so @Test(expectedExceptions = "...")
> is of less use for negative testing.
> 4.      Also we want to keep the test log in a separate file for each test
> so that we can refer to it at a later point of time and should have an
> hyperlink to this file from the generated report.
>
> So Please let me know is it feasible to use TestNG for above
> requirements?

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@...
To unsubscribe from this group, send email to testng-users+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[John A Thompson]

I think TestNg will cover most of what you are looking for.  It has some very robust options you can work with.

For setup, I suggest you look at assigning groups at the class level.  This can be done by using the @Test annotation at the class level.  All the groups in the class will inherit the group.

What I've found is easy for organization of test is to use packages to define larger groups.  The packages of tests can then be run as suites via the testng.xml configuration. (allowing you to run groups of groups).  Its easier and more dynamic to specify running all classes in a package, vs assigning individual classes.  Or you could use groups, but it sounded like you thought this would be too difficult to maintain.

The HTML reports do a nice job of keeping the logging separate. (I'm building in Maven, and could not set the file fragementation level to get the logging split out for the xml reports.)

For your #3 requirement, not sure if there is a way to make the exception conditional.  You might have to cook this logic into your test class.

On Mon, Nov 10, 2008 at 3:53 AM, kiri <kiran.eshwarappa@...> wrote:

I was evaluating TestNG to use it in one of my projects to achieve
functional testing and stress testing. What I discovered are the
following things.

1.      We treat the whole Class as a functional test which includes
setup(), test() cleanup() and some helper methods so there will be
only one method in super class 'executeTest()' which needs to be
called, And the Test Class many be at 4 level down the hierarchy as
tests needs complex code which are modularized. So it is not possible
to tag individual tests with different groups (we need to override
executeTest() in each Class to do this) as we are going to write
around 1000 tests it's not manageable if the test groups needs to be
modified frequently for many number of tests.

2.      Lets say I have to tag the tests with
Product versions (PV01, PV02, PV03, PV04)
Applicable clients (C1,C2,C3,C4) and many more

So I want a way to configure the mapping between Test Class and the
applicable tags in single place
Ex: com.module.submodule.Pos001 = PV01,PV02,C1,C4,
com.module.submodule.Pos002 = PV01,C1,C2,C3. in a property file, and
generate the xmlSuite for suites (PV01 && C1) , (PV02 && C2) etc and
run the generated suites through TestNg. So that I have fine grain
control on what I am going to run. Here I am not tagging the tests but
need to make sure that the test belongs to groups as specified in
property file dynamically. Now let's say 20 tests are also applicable
for C5, I just need to modify this file. Is there a way to do it?

3.      There will be many tests which will be applicable for many
combination of product and clients and the Exceptions depends on the
product on which we run the test, so @Test(expectedExceptions = "...")
is of less use for negative testing.
4.      Also we want to keep the test log in a separate file for each test
so that we can refer to it at a later point of time and should have an
hyperlink to this file from the generated report.

So Please let me know is it feasible to use TestNG for above
requirements?

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@...
To unsubscribe from this group, send email to testng-users+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[Cédric Beust ♔]

Hi Kiran,

From what I understand in your requirements, you need a lot of dynamicity in the way you want to group your tests, so you are probably going to use some of the more advanced features of TestNG, among which:

• Generation of XML files with the org.testng.xml API (they don't have to be physical files by the way, they can be XMLSuite objects that you pass directly to the TestNG runner).
  
  
• IAnnotationTransformer so that groups can be calculated and assigned at runtime instead of statically.

Hope this helps.

--

Cédric

On Mon, Nov 10, 2008 at 12:53 AM, kiri <kiran.eshwarappa@...> wrote:

I was evaluating TestNG to use it in one of my projects to achieve
functional testing and stress testing. What I discovered are the
following things.

1.      We treat the whole Class as a functional test which includes
setup(), test() cleanup() and some helper methods so there will be
only one method in super class 'executeTest()' which needs to be
called, And the Test Class many be at 4 level down the hierarchy as
tests needs complex code which are modularized. So it is not possible
to tag individual tests with different groups (we need to override
executeTest() in each Class to do this) as we are going to write
around 1000 tests it's not manageable if the test groups needs to be
modified frequently for many number of tests.

2.      Lets say I have to tag the tests with
Product versions (PV01, PV02, PV03, PV04)
Applicable clients (C1,C2,C3,C4) and many more

So I want a way to configure the mapping between Test Class and the
applicable tags in single place
Ex: com.module.submodule.Pos001 = PV01,PV02,C1,C4,
com.module.submodule.Pos002 = PV01,C1,C2,C3. in a property file, and
generate the xmlSuite for suites (PV01 && C1) , (PV02 && C2) etc and
run the generated suites through TestNg. So that I have fine grain
control on what I am going to run. Here I am not tagging the tests but
need to make sure that the test belongs to groups as specified in
property file dynamically. Now let's say 20 tests are also applicable
for C5, I just need to modify this file. Is there a way to do it?

3.      There will be many tests which will be applicable for many
combination of product and clients and the Exceptions depends on the
product on which we run the test, so @Test(expectedExceptions = "...")
is of less use for negative testing.
4.      Also we want to keep the test log in a separate file for each test
so that we can refer to it at a later point of time and should have an
hyperlink to this file from the generated report.

So Please let me know is it feasible to use TestNG for above
requirements?

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@...
To unsubscribe from this group, send email to testng-users+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---