[jira] Created: (DERBY-3780) Run junit tests with -Dfile.encoding="UTF-16" to expose encoding issues and analyze failures

Run junit tests with -Dfile.encoding="UTF-16" to expose encoding issues and analyze failures
--------------------------------------------------------------------------------------------

                 Key: DERBY-3780
                 URL: https://issues.apache.org/jira/browse/DERBY-3780
             Project: Derby
          Issue Type: Task
          Components: Test
    Affects Versions: 10.5.0.0
            Reporter: Kathey Marsden
            Priority: Minor


Wth sun jkd 1.5 and 1.6 you can run JUnit tests with -Dfile.encoding="UTF-16" to help expose encoding issues with Derby and tests.    Run suites.All and analyze failures.



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DERBY-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kathey Marsden updated DERBY-3780:
----------------------------------

    Attachment: ReadEncodedFile.java
                suites_all_utf16.txt

Here are the unalyzed results of a run of suiltes.all with -Dfile.encoding=UTF-16.  Interestingly there are some failures that did not show up on Zos. Also attaching a utility ReadEncodedFile which I had to use to translate the UTF-16 output into the default encoding.

> Run junit tests with -Dfile.encoding="UTF-16" to expose encoding issues and analyze failures
> --------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3780
>                 URL: https://issues.apache.org/jira/browse/DERBY-3780
>             Project: Derby
>          Issue Type: Task
>          Components: Test
>    Affects Versions: 10.5.0.0
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: ReadEncodedFile.java, suites_all_utf16.txt
>
>
> Wth sun jkd 1.5 and 1.6 you can run JUnit tests with -Dfile.encoding="UTF-16" to help expose encoding issues with Derby and tests.    Run suites.All and analyze failures.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614680#action_12614680 ]

Knut Anders Hatlen commented on DERBY-3780:
-------------------------------------------

I haven't studied the results in detail, I just wanted to add that the file.encoding property is not a standard property, and it is meant to be a read-only property that tells you the default encoding. Since you're not supposed to change the property, some parts of the class library may use it to determine the default encoding, whereas others may use another way to find it. So some of the failures may be caused by the class library being confused and using the encoding inconsistently.

Seems like you also ran out of perm gen space. Possibly because UTF-16 requires more space in many cases?

> Run junit tests with -Dfile.encoding="UTF-16" to expose encoding issues and analyze failures
> --------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3780
>                 URL: https://issues.apache.org/jira/browse/DERBY-3780
>             Project: Derby
>          Issue Type: Task
>          Components: Test
>    Affects Versions: 10.5.0.0
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: ReadEncodedFile.java, suites_all_utf16.txt
>
>
> Wth sun jkd 1.5 and 1.6 you can run JUnit tests with -Dfile.encoding="UTF-16" to help expose encoding issues with Derby and tests.    Run suites.All and analyze failures.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614799#action_12614799 ]

Kathey Marsden commented on DERBY-3780:
---------------------------------------

Thanks Knut for looking at this. Below is my analysis.

--------------------------------------------------------------------------------------------
ClobTest,  BlobClob4Blob and LargeDataLocks tests assume length of string inserted into a binary column is equal to the length of the bytes that represent the string.  These did not fail on Zos because even though the values are different for EBCDIC, the alphabet characters still only take one byte.  The tests should be changed to not make this assumption.

1) testGetCharacterStreamClobUpdates(org.apache.derbyTesting.functionTests.tests.jdbc4.ClobTest)java.sql.SQLException: The position argument '53' exceeds the size of the BLOB/CLOB.

16) testPositionBlob(org.apache.derbyTesting.functionTests.tests.jdbcapi.BlobClob4BlobTest)junit.framework.AssertionFailedError: FAIL - wrong match found for ??? starting at 52 and length of 3 expected:<57> but was:<-1>

17) testGetBlobBeforeAndAfterUpdate(org.apache.derbyTesting.functionTests.tests.jdbcapi.BlobClob4BlobTest)junit.framework.AssertionFailedError: FAIL - wrong blob value

18) testGetBlobBeforeAndAfterUpdateStream(org.apache.derbyTesting.functionTests.tests.jdbcapi.BlobClob4BlobTest)junit.framework.AssertionFailedError: FAIL - wrong blob value

19) testPositionBytes(org.apache.derbyTesting.functionTests.tests.jdbcapi.BlobClob4BlobTest)junit.framework.AssertionFailedError: FAIL - wrong match found for ? starting at 40 and length of 1 expected:<62> but was:<47>

20) testPositionBlob(org.apache.derbyTesting.functionTests.tests.jdbcapi.Blob
Clob4BlobTest)junit.framework.AssertionFailedError: FAIL - wrong match found for ????????? starting at 38 and length of 9 expected:<39> but was:<-1>

27) testGetBytes(org.apache.derbyTesting.functionTests.tests.jdbcapi.LargeDataLocksTest)junit.framework.AssertionFailedError: expected:<38000> but was:<76002>
        at org.apache.derbyTesting.functionTests.tests.jdbcapi.LargeDataLocksTest.te

26) testGetBinaryStream(org.apache.derbyTesting.functionTests.tests.jdbcapi.LargeDataLocksTest)junit.framew
ork.AssertionFailedError: expected:<38000> but was:<76002>
------------------------------------------------------------------------
File not imported with US-ASCII codeset. Test needs to be fixed.

30) testEarlyEndOfFile(org.apache.derbyTesting.functionTests.tests.tools.ImportExportTest)junit.framework.ComparisonFailure: Unexpected SQL state. expected:<XIE0E> but was:<42X04>
--------------------------------------------------------------------
Tests that spawn processes that don't have the same file.encoding.  Can't really fix this without passing the file.encoding on to the spawned process which we probably don't want to do since it is not a standard property.

2) SecureServerTest( Opened = false, Authenticated= false, CustomDerbyProperties= null, WildCardHost= null )junit.framework.AssertionFailedError: SecureServerTest( Opened = false, Authenticated= false, CustomDerbyProperties= null, WildCardHost= null )
Expected: Security manager installed using the Basic server security policy.
But saw: ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
----------------------------------------------------------------------------
Problem reading error log vti.  The support file is in ascii and the SYSCS_DIAG.ERROR_LOG_READER vti I think correctly reads in the default encoding, so I don't think this is a problem.  Not sure why this one didn't show up on Zos.


13) testErrorLogReader(org.apache.derbyTesting.functionTests.tests.lang.SysDiagVTIMappingTest)junit.framework.AssertionFailedError: Unexpected row count: expected:<2> but was:<0>
---------------------------------------------------------------------------
Permgen Space errors and cascading DatabaseMetaDataTest failures.  This just occured because I forgot to up the Permgen space when running with the sun jvm.

-------------------------------------------------------------------
Pemission errors getting context class loader in DropDatabaseTestSetup.  Not sure what could be causing this or even what test is causing the problem.  Maybe an encoding problem reading the policy file, but only a few tests seemed to have this problem.

19) Encryption Algorithm: defaultjava.security.AccessControlException: access denied (java.lang.RuntimePermission getClassLoader)

--------------------------------------------------------------------------------------------------------------------
In general the conclusion is running this way popped a few more test encoding issues and this is a nice trick to reproduce encoding issues on Windows with Sun JVM's (doesn't work with IBM jvms).  I don't see any product encoding issues.  Running with -Dfile.encoding=UTF-16  it is not really suitable for a nightly run, which I had hoped, because of the spawned processes and AccessControl exceptions.

> Run junit tests with -Dfile.encoding="UTF-16" to expose encoding issues and analyze failures
> --------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3780
>                 URL: https://issues.apache.org/jira/browse/DERBY-3780
>             Project: Derby
>          Issue Type: Task
>          Components: Test
>    Affects Versions: 10.5.0.0
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: ReadEncodedFile.java, suites_all_utf16.txt
>
>
> Wth sun jkd 1.5 and 1.6 you can run JUnit tests with -Dfile.encoding="UTF-16" to help expose encoding issues with Derby and tests.    Run suites.All and analyze failures.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DERBY-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kathey Marsden updated DERBY-3780:
----------------------------------

    Attachment: derby-3780_diff.txt

Attached is a patch to fix the test encoding issues with
jdbc4/ClobTest.java
jdbcapi/LargeDataLocksTest.java
jdbcapi/BlobClob4BlobTest.java
tools/ImportExportTest.java

> Run junit tests with -Dfile.encoding="UTF-16" to expose encoding issues and analyze failures
> --------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3780
>                 URL: https://issues.apache.org/jira/browse/DERBY-3780
>             Project: Derby
>          Issue Type: Task
>          Components: Test
>    Affects Versions: 10.5.0.0
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: derby-3780_diff.txt, ReadEncodedFile.java, suites_all_utf16.txt
>
>
> Wth sun jkd 1.5 and 1.6 you can run JUnit tests with -Dfile.encoding="UTF-16" to help expose encoding issues with Derby and tests.    Run suites.All and analyze failures.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DERBY-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614887#action_12614887 ]

Knut Anders Hatlen commented on DERBY-3780:
-------------------------------------------

The changes look good to me. You may want to change the code in the decorateSQL() method in LargeDataLocksTest so that the UnsupportedEncodingException is thrown, not just printed. decorateSQL() cannot throw other checked exceptions than SQLException, but you can use BaseTestCase.fail(String, Exception) to get around that problem.

> Run junit tests with -Dfile.encoding="UTF-16" to expose encoding issues and analyze failures
> --------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3780
>                 URL: https://issues.apache.org/jira/browse/DERBY-3780
>             Project: Derby
>          Issue Type: Task
>          Components: Test
>    Affects Versions: 10.5.0.0
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: derby-3780_diff.txt, ReadEncodedFile.java, suites_all_utf16.txt
>
>
> Wth sun jkd 1.5 and 1.6 you can run JUnit tests with -Dfile.encoding="UTF-16" to help expose encoding issues with Derby and tests.    Run suites.All and analyze failures.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DERBY-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kathey Marsden closed DERBY-3780.
---------------------------------

    Resolution: Fixed
      Assignee: Kathey Marsden

Ran tests and found some test encoding issues which I corrected.

> Run junit tests with -Dfile.encoding="UTF-16" to expose encoding issues and analyze failures
> --------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3780
>                 URL: https://issues.apache.org/jira/browse/DERBY-3780
>             Project: Derby
>          Issue Type: Task
>          Components: Test
>    Affects Versions: 10.5.0.0
>            Reporter: Kathey Marsden
>            Assignee: Kathey Marsden
>            Priority: Minor
>         Attachments: derby-3780_diff.txt, ReadEncodedFile.java, suites_all_utf16.txt
>
>
> Wth sun jkd 1.5 and 1.6 you can run JUnit tests with -Dfile.encoding="UTF-16" to help expose encoding issues with Derby and tests.    Run suites.All and analyze failures.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DERBY-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kathey Marsden updated DERBY-3780:
----------------------------------

    Fix Version/s: 10.5.0.0

> Run junit tests with -Dfile.encoding="UTF-16" to expose encoding issues and analyze failures
> --------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3780
>                 URL: https://issues.apache.org/jira/browse/DERBY-3780
>             Project: Derby
>          Issue Type: Task
>          Components: Test
>    Affects Versions: 10.5.0.0
>            Reporter: Kathey Marsden
>            Assignee: Kathey Marsden
>            Priority: Minor
>             Fix For: 10.5.0.0
>
>         Attachments: derby-3780_diff.txt, ReadEncodedFile.java, suites_all_utf16.txt
>
>
> Wth sun jkd 1.5 and 1.6 you can run JUnit tests with -Dfile.encoding="UTF-16" to help expose encoding issues with Derby and tests.    Run suites.All and analyze failures.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.