Testing JMagick

[Jacob Nordfalk]

Hi all,

My proposed JUnit testing scheme for JMagick is ready.
Please comment if you disagree on any of these things. If no
objections are made in 5 days I will consider all the below APPROVED
by the jmagick community.

For those that don't know about Unit testing, the idea is to write
some methods that use JMagick in different ways and then to compare
the result against expected values and report an error if the results
are not as expected. See also http://junit.org/ and
http://junit.sourceforge.net/junit3.8.1/index.html .

For JMagick I propose
1) we use JUnit 3 (that is, JDK1.4 no fancy new things from JDK1.5).
2) we maintain a collection of 'correct output' images to compare with
in 'test/correct_output'
3) all accepted tests are in package 'magicktest'. All
new/proposed/not finished/not yet accepted tests are in package
'magicktest.BETA'
4) the goal is to get 100% code coverage, that is: Running the test
suite should activate all JMagick methods.

When 100% code coverage is acieved is will be muuuch easier and safer
to edit the source code of JMagick:
* any errors introduced will show up in the tests (probably, at least :-).
* Any new incompatibilities becaurse of IM code change will
immediately be found.
* we can safely (without breaking anything) move to other ways of
usingIM/GraphicsMagick functions from Java and keep compatibility
(I have automatick exposure/code generation usning JNA in mind -
others consider using the much easier C++ API).



In test/magicktest is the test classes. All test classes are here, and
not in src/ anymore.

Eric's old simpe 'test' (Test.java - but it's really more some
examples of usage than a test) have been renamed to
'OldSimpleTest.java'

Eric's work has been convertet to a JUnit test case in
TestJMagick.java. Please have a look now before you continue reading:
http://jmagick.svn.sourceforge.net/viewvc/jmagick/test/magicktest/TestJMagick.java?view=markup

Here in an example:

   65 ImageInfo info = new ImageInfo(Testtools.path_input + "pics.jpg");
...
   75 image = new MagickImage(info);
...
   82 assertEquals("X resolution is ", 72, image.getXResolution(), 0.1);
   83 assertEquals("Y resolution is ", 72, image.getYResolution(), 0.1);

The methods assertEquals() CHECKS that the return value of
image.getYResolution() is 72  (+/- 0.1) and if this is not the case an
exetion is thrown and the test fails.
Now: THAT is (unit) testing.


Here in another example:

  116 public void testRaise() throws Exception {
  117 // Raise image
  118 image.raiseImage(new Rectangle(0, 0, 10, 20), true);
  119 MagickTesttools.writeAndCompare(image, info, "raised.jpg");
  120 }

Here we use a utility method MagickTesttools which writes the image in
the directory test/actual_output ("test/actual_output/raised.jpg") and
then compares this image with "test/correct_output/raised.jpg". If any
differences are found (within a small margin) then an exception is
thrown and the test fails.


Please check out/update from SVN and try it!
UNIX command line users should be able to just wrtie 'make test'  (but
you need to install junit.jar in lib/ directory).
Windows/GUI users should configure their project to include JUnit
testing and run the class TestJMagick.
Please report any problems you may have.

In test/correct_output is images as they are expected to be. If you
have made an update to JMagick and wants to generate a new set of
'correct' output images you should 'make generateCorrect' or run the
class GenerateCorrectOutput and follow the instructions.


I look forward to hear your reactions!

I hope everyone will try the new test suite on as many platforms as
possible and report any success/failure.

Some tests are dependent on the fonts installed on the system and will
thus probably fail. These have been '_failMightBeOk' in their name (to
not upset you).



--
Jacob Nordfalk
http://javabog.dk
Skype: jacobnordfalk
_______________________________________________
JMagick mailing list
JMagick@...
http://www.yeo.id.au/mailman/listinfo/jmagick
Please ensure that JMagick@... is a recipient when replying.

[Bob Friesenhahn]

On Thu, 28 Feb 2008, Jacob Nordfalk wrote:
>
> For those that don't know about Unit testing, the idea is to write
> some methods that use JMagick in different ways and then to compare
> the result against expected values and report an error if the results
> are not as expected. See also http://junit.org/ and
> http://junit.sourceforge.net/junit3.8.1/index.html .

I have been through this a number of times before, having written the
test suites for ImageMagick, GraphicsMagick, and Magick++.  When
comparing output images against a reference image, you really need to
do a statisical comparison rather than absolute.  Otherwise
differences between Q8/Q16/Q32 will cause test failure.  Formats like
JPEG are lossy so they require a large "fudge factor".  I think that
reference images should be generated using a Q8 build since they
represent the maximum variation from the "one true image" and the
files are likely to be smaller.  Dealing with fonts is a major problem
since each version of FreeType produces different results, and glyph
hinting (FreeType compile-time options) can make a huge difference in
the output.  There are similar problems with PDF and Postscript.

Bob
======================================
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

_______________________________________________
JMagick mailing list
JMagick@...
http://www.yeo.id.au/mailman/listinfo/jmagick
Please ensure that JMagick@... is a recipient when replying.

[Jacob Nordfalk]

>
> I have been through this a number of times before, having written the
>  test suites for ImageMagick, GraphicsMagick, and Magick++.  When
>  comparing output images against a reference image, you really need to
>  do a statisical comparison rather than absolute.

Yes. If you look in MagickTesttools line 87 you wil se that that's
exactly what is being done:
(http://jmagick.svn.sourceforge.net/viewvc/jmagick/test/magicktest/MagickTesttools.java?view=markup)

                String cmd = "compare -metric MSE "+file1+" "+file2+"
tempImageDifference.jpg";


So 'compare -metric MSE.correctFile.jpg actualFile.jpg' is being used
to determine the difference.

I have used this technique the last year and my experience is this:
Usually differences are under 1 on the test image and images supposed
to be identical
but I have set a "fudge factor" margin  of 20 for the maximal
statistical derivation.

When images differ becaurse of algorithmic errors, they usually do
this with aprox 100  or more (for example input pics.jpg and
sharpened.jpg differs by 77, convolved.jpg by 70 and quantized.png by
943).


> Otherwise
>  differences between Q8/Q16/Q32 will cause test failure.

I dont think so, for the above reasons. But all has been done with Q8
only, I still needs someone to try on Q16.


> Formats like
>  JPEG are lossy so they require a large "fudge factor".

Yes. Therefore the standard "fudge factor" margin is set to 20
(usually a margin of 2 is really enough).


> I think that
>  reference images should be generated using a Q8 build since they
>  represent the maximum variation from the "one true image" and the
>  files are likely to be smaller.

I agree and that's also how Ive done it.


>  Dealing with fonts is a major problem
>  since each version of FreeType produces different results, and glyph
>  hinting (FreeType compile-time options) can make a huge difference in
>  the output.  There are similar problems with PDF and Postscript.

I agree and that's also how Ive done it:
Some tests are dependent on the fonts installed on the system and will
thus probably fail. These have been '_failMightBeOk' in their name (to
not upset you).



--
Jacob Nordfalk
http://javabog.dk
Skype: jacobnordfalk
_______________________________________________
JMagick mailing list
JMagick@...
http://www.yeo.id.au/mailman/listinfo/jmagick
Please ensure that JMagick@... is a recipient when replying.

[Terence M. Bandoian]

Hi-

Does 1) mean not testing against JDK1.5 at all?

-Terence M. Bandoian

> Date: Thu, 28 Feb 2008 11:55:12 +0545
> From: "Jacob Nordfalk" <jacob.nordfalk@...>
> Subject: [Jmagick] Testing JMagick
> To: jmagick@...
>
> Hi all,
>
> My proposed JUnit testing scheme for JMagick is ready.
> Please comment if you disagree on any of these things. If no
> objections are made in 5 days I will consider all the below APPROVED
> by the jmagick community.
>
> For those that don't know about Unit testing, the idea is to write
> some methods that use JMagick in different ways and then to compare
> the result against expected values and report an error if the results
> are not as expected. See also http://junit.org/ and
> http://junit.sourceforge.net/junit3.8.1/index.html .
>
> For JMagick I propose
> 1) we use JUnit 3 (that is, JDK1.4 no fancy new things from JDK1.5).
> 2) we maintain a collection of 'correct output' images to compare with
> in 'test/correct_output'
> 3) all accepted tests are in package 'magicktest'. All
> new/proposed/not finished/not yet accepted tests are in package
> 'magicktest.BETA'
> 4) the goal is to get 100% code coverage, that is: Running the test
> suite should activate all JMagick methods.
>
> When 100% code coverage is acieved is will be muuuch easier and safer
> to edit the source code of JMagick:
> * any errors introduced will show up in the tests (probably, at least :-).
> * Any new incompatibilities becaurse of IM code change will
> immediately be found.
> * we can safely (without breaking anything) move to other ways of
> usingIM/GraphicsMagick functions from Java and keep compatibility
> (I have automatick exposure/code generation usning JNA in mind -
> others consider using the much easier C++ API).
>
>
>
> In test/magicktest is the test classes. All test classes are here, and
> not in src/ anymore.
>
> Eric's old simpe 'test' (Test.java - but it's really more some
> examples of usage than a test) have been renamed to
> 'OldSimpleTest.java'
>
> Eric's work has been convertet to a JUnit test case in
> TestJMagick.java. Please have a look now before you continue reading:
> http://jmagick.svn.sourceforge.net/viewvc/jmagick/test/magicktest/TestJMagick.java?view=markup
>
> Here in an example:
>
>    65 ImageInfo info = new ImageInfo(Testtools.path_input + "pics.jpg");
> ...
>    75 image = new MagickImage(info);
> ...
>    82 assertEquals("X resolution is ", 72, image.getXResolution(), 0.1);
>    83 assertEquals("Y resolution is ", 72, image.getYResolution(), 0.1);
>
> The methods assertEquals() CHECKS that the return value of
> image.getYResolution() is 72  (+/- 0.1) and if this is not the case an
> exetion is thrown and the test fails.
> Now: THAT is (unit) testing.
>
>
> Here in another example:
>
>   116 public void testRaise() throws Exception {
>   117 // Raise image
>   118 image.raiseImage(new Rectangle(0, 0, 10, 20), true);
>   119 MagickTesttools.writeAndCompare(image, info, "raised.jpg");
>   120 }
>
> Here we use a utility method MagickTesttools which writes the image in
> the directory test/actual_output ("test/actual_output/raised.jpg") and
> then compares this image with "test/correct_output/raised.jpg". If any
> differences are found (within a small margin) then an exception is
> thrown and the test fails.
>
>
> Please check out/update from SVN and try it!
> UNIX command line users should be able to just wrtie 'make test'  (but
> you need to install junit.jar in lib/ directory).
> Windows/GUI users should configure their project to include JUnit
> testing and run the class TestJMagick.
> Please report any problems you may have.
>
> In test/correct_output is images as they are expected to be. If you
> have made an update to JMagick and wants to generate a new set of
> 'correct' output images you should 'make generateCorrect' or run the
> class GenerateCorrectOutput and follow the instructions.
>
>
> I look forward to hear your reactions!
>
> I hope everyone will try the new test suite on as many platforms as
> possible and report any success/failure.
>
> Some tests are dependent on the fonts installed on the system and will
> thus probably fail. These have been '_failMightBeOk' in their name (to
> not upset you).

_______________________________________________
JMagick mailing list
JMagick@...
http://www.yeo.id.au/mailman/listinfo/jmagick
Please ensure that JMagick@... is a recipient when replying.

[Jacob Nordfalk]

Now almost a month have passed (so my proposal is definitely approved :-).

I would very much like some feedback from more of you:

1) Did you try the tests?
2) Did they run OK? (please send error reports)
3) Do you understand the principle? Would it be easy enough for you to
contribute with some tests?

Jacob


2008/2/28, Jacob Nordfalk <jacob.nordfalk@...>:

> Hi all,
>
>  My proposed JUnit testing scheme for JMagick is ready.
>  Please comment if you disagree on any of these things. If no
>  objections are made in 5 days I will consider all the below APPROVED
>  by the jmagick community.
>
>  For those that don't know about Unit testing, the idea is to write
>  some methods that use JMagick in different ways and then to compare
>  the result against expected values and report an error if the results
>  are not as expected. See also http://junit.org/ and
>  http://junit.sourceforge.net/junit3.8.1/index.html .
>
>  For JMagick I propose
>  1) we use JUnit 3 (that is, JDK1.4 no fancy new things from JDK1.5).
>  2) we maintain a collection of 'correct output' images to compare with
>  in 'test/correct_output'
>  3) all accepted tests are in package 'magicktest'. All
>  new/proposed/not finished/not yet accepted tests are in package
>  'magicktest.BETA'
>  4) the goal is to get 100% code coverage, that is: Running the test
>  suite should activate all JMagick methods.
>
>  When 100% code coverage is acieved is will be muuuch easier and safer
>  to edit the source code of JMagick:
>  * any errors introduced will show up in the tests (probably, at least :-).
>  * Any new incompatibilities becaurse of IM code change will
>  immediately be found.
>  * we can safely (without breaking anything) move to other ways of
>  usingIM/GraphicsMagick functions from Java and keep compatibility
>  (I have automatick exposure/code generation usning JNA in mind -
>  others consider using the much easier C++ API).
>
>
>
>  In test/magicktest is the test classes. All test classes are here, and
>  not in src/ anymore.
>
>  Eric's old simpe 'test' (Test.java - but it's really more some
>  examples of usage than a test) have been renamed to
>  'OldSimpleTest.java'
>
>  Eric's work has been convertet to a JUnit test case in
>  TestJMagick.java. Please have a look now before you continue reading:
>  http://jmagick.svn.sourceforge.net/viewvc/jmagick/test/magicktest/TestJMagick.java?view=markup
>
>  Here in an example:
>
>    65           ImageInfo info = new ImageInfo(Testtools.path_input + "pics.jpg");
>  ...
>    75           image = new MagickImage(info);
>  ...
>    82           assertEquals("X resolution is ", 72, image.getXResolution(), 0.1);
>    83           assertEquals("Y resolution is ", 72, image.getYResolution(), 0.1);
>
>  The methods assertEquals() CHECKS that the return value of
>  image.getYResolution() is 72  (+/- 0.1) and if this is not the case an
>  exetion is thrown and the test fails.
>  Now: THAT is (unit) testing.
>
>
>  Here in another example:
>
>   116   public void testRaise() throws Exception {
>   117           // Raise image
>   118           image.raiseImage(new Rectangle(0, 0, 10, 20), true);
>   119           MagickTesttools.writeAndCompare(image, info, "raised.jpg");
>   120   }
>
>  Here we use a utility method MagickTesttools which writes the image in
>  the directory test/actual_output ("test/actual_output/raised.jpg") and
>  then compares this image with "test/correct_output/raised.jpg". If any
>  differences are found (within a small margin) then an exception is
>  thrown and the test fails.
>
>
>  Please check out/update from SVN and try it!
>  UNIX command line users should be able to just wrtie 'make test'  (but
>  you need to install junit.jar in lib/ directory).
>  Windows/GUI users should configure their project to include JUnit
>  testing and run the class TestJMagick.
>  Please report any problems you may have.
>
>  In test/correct_output is images as they are expected to be. If you
>  have made an update to JMagick and wants to generate a new set of
>  'correct' output images you should 'make generateCorrect' or run the
>  class GenerateCorrectOutput and follow the instructions.
>
>
>  I look forward to hear your reactions!
>
>  I hope everyone will try the new test suite on as many platforms as
>  possible and report any success/failure.
>
>  Some tests are dependent on the fonts installed on the system and will
>  thus probably fail. These have been '_failMightBeOk' in their name (to
>  not upset you).
>
>
>
>
>  --
>  Jacob Nordfalk
>  http://javabog.dk
>  Skype: jacobnordfalk
>

--
Jacob Nordfalk
http://javabog.dk
Skype: jacobnordfalk
_______________________________________________
JMagick mailing list
JMagick@...
http://www.yeo.id.au/mailman/listinfo/jmagick
Please ensure that JMagick@... is a recipient when replying.