i18n/JapanCodeConversion test intermittent failures & changes

[Daniel John Debrunner <dj...@debrunners.com>]

With the i18n/JapanCodeConversion test I see failures due to the
execution of native2ascii failing. This executing an external process
from ij also causes problems running the test under a security manager.

The test exports data with Japanese unicode characters from a table into
two files, one using EUC_JP encoding and the other SJIS encoding.

The content of these files are "checked for correctness" by dumping the
output of the JDK utility native2ascii to System.out and hence to the
master file.

I've added additional tests that better test the correctness of the
exported data by:

 - importing the exported data using EUC_JP and SJIS into separate
tables and comparing the data to the original table's data. This
actually checks the data is not modified by the export/import process
with the two encodings.

 - importing the data using EUC_JP and SJIS, but from the wrong files
(ie. import from the SJIS file using EUC_JP encoding) and comparing with
the original data. This ensures that a single format is not being used.

This was checked in with svn revision 348553.

Do these changes seem sufficient to remove the nasty use of the
executable native2ascii for this test?

Dan.

Re: i18n/JapanCodeConversion test intermittent failures & changes

[Daniel John Debrunner <dj...@debrunners.com>]

TomohitoNakayama wrote:

> Hello.
> 
> I think checking only finally imported data at dbms and no check at file
> of native encoding will conceal problem if there exists symmetric
> problem in import/export ....
> / /I mean that the combination of wrong behaviors ,of import and export
> ,would confuse us to think they work correct ...
> 

Right, that's what I was trying to do with the importing with the wrong
encoding.

However I think I can easily mimic the native2ascii program using Java
procedures/functions,
thus removing the issues caused by exec'ing native2ascii.

Thanks,
Dan.

Re: i18n/JapanCodeConversion test intermittent failures & changes

[TomohitoNakayama <to...@basil.ocn.ne.jp>]

Hello.

I think checking only finally imported data at dbms and no check at file 
of native encoding will conceal problem if there exists symmetric 
problem in import/export ....
/ /I mean that the combination of wrong behaviors ,of import and export 
,would confuse us to think they work correct ...

My idea is to import data as binary to dbms and check it as numeric data .

One possible problem is that there may exist difference of 
character-converting table between platforms in some characters ....
Next is example of such a problem found in Oracle . You will see what 
kind of character would be problematic .
    
http://www.oracle.com/technology/tech/java/sqlj_jdbc/htdocs/nlsalert.html

If this kind of problem happens, binary data in exported file may be 
different between platforms ....

I think our test can ignore this problem of different 
character-converting table , not using character which would cause 
difference of converting table problem,
because we are not developing Unicode character-converting table ....

// I wish someone realize uni-character-converting table of Unicode 
across all vendors :)

Best regards .


Daniel John Debrunner wrote:

>With the i18n/JapanCodeConversion test I see failures due to the
>execution of native2ascii failing. This executing an external process
>from ij also causes problems running the test under a security manager.
>
>The test exports data with Japanese unicode characters from a table into
>two files, one using EUC_JP encoding and the other SJIS encoding.
>
>The content of these files are "checked for correctness" by dumping the
>output of the JDK utility native2ascii to System.out and hence to the
>master file.
>
>I've added additional tests that better test the correctness of the
>exported data by:
>
> - importing the exported data using EUC_JP and SJIS into separate
>tables and comparing the data to the original table's data. This
>actually checks the data is not modified by the export/import process
>with the two encodings.
>
> - importing the data using EUC_JP and SJIS, but from the wrong files
>(ie. import from the SJIS file using EUC_JP encoding) and comparing with
>the original data. This ensures that a single format is not being used.
>
>This was checked in with svn revision 348553.
>
>Do these changes seem sufficient to remove the nasty use of the
>executable native2ascii for this test?
>
>Dan.
>
>
>
>  
>

-- 
/*

        Tomohito Nakayama
        tomonaka@basil.ocn.ne.jp
        tomohito@rose.zero.ad.jp
        tmnk@apache.org

        Naka
        http://www5.ocn.ne.jp/~tomohito/TopPage.html

*/