[jira] [Commented] (HADOOP-10744) LZ4 Compression fails to recognize PowerPC Little Endian Architecture

["Ayappan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-10744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041968#comment-14041968 ] 

Ayappan commented on HADOOP-10744:
----------------------------------

The problem is due to the below lines of code in the lz4.c file (src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c)

// Little Endian or Big Endian ?
// Overwrite the #define below if you know your architecture endianess
#if defined (__GLIBC__)
#  include <endian.h>
#  if (__BYTE_ORDER == __BIG_ENDIAN)
#     define LZ4_BIG_ENDIAN 1
#  endif
#elif (defined(__BIG_ENDIAN__) || defined(__BIG_ENDIAN) || defined(_BIG_ENDIAN)) && !(defined(__LITTLE_ENDIAN__) || defined(__LITTLE_ENDIAN) || defined(_LITTLE_ENDIAN))
#  define LZ4_BIG_ENDIAN 1
#elif defined(__sparc) || defined(__sparc__) \
   || defined(__powerpc__) || defined(__ppc__) || defined(__PPC__) \
   || defined(__hpux)  || defined(__hppa) \
   || defined(_MIPSEB) || defined(__s390__)
#  define LZ4_BIG_ENDIAN 1
#else
// Little Endian assumed. PDP Endian and other very rare endian format are unsupported.
#endif

The flow goes like this in Power little Endian system. __GLIB__ is not defined , it falls on next elif. The condition fails due to __LITTLE_ENDIAN__... flags, falls on next elif and here it get satisfied! and sets LZ4_BIG_ENDIAN. The thing is these flags __powerpc__, __ppc__, __PPC__ are also defined in PowerPC Little Endian Architecture.

There is a similar file called lz4hc.c which also has the same lines code and still it correctly recognizes Little Endian Architecture. The reason behind that is..there is a include <stdlib.h> which actually turns on the __GLIBC__ flag.
And since now the decision is based on BYTE_ORDER, it identifies whether it is Big Endian or Little Endian.

So lz4.c file should also have the "include <stdlib.h>" to make it work correctly in Power Little Endian Architecture


> LZ4 Compression fails to recognize PowerPC Little Endian Architecture
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-10744
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10744
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io, native
>    Affects Versions: 2.2.0, 2.3.0, 2.4.0
>         Environment: PowerPC Little Endian (ppc64le)
>            Reporter: Ayappan
>         Attachments: HADOOP-10744.patch
>
>
> Lz4 Compression fails to identify the PowerPC Little Endian Architecture. It recognizes it as Big Endian and several testcases( TestCompressorDecompressor, TestCodec, TestLz4CompressorDecompressor)  fails due to this.
> Running org.apache.hadoop.io.compress.TestCompressorDecompressor
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.435 sec <<< FAILURE! - in org.apache.hadoop.io.compress.TestCompressorDecompressor
> testCompressorDecompressor(org.apache.hadoop.io.compress.TestCompressorDecompressor)  Time elapsed: 0.308 sec  <<< FAILURE!
> org.junit.internal.ArrayComparisonFailure: org.apache.hadoop.io.compress.lz4.Lz4Compressor_org.apache.hadoop.io.compress.lz4.Lz4Decompressor-  byte arrays not equals error !!!: arrays first differed at element [1428]; expected:<4> but was:<10>
>         at org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
>         at org.junit.Assert.internalArrayEquals(Assert.java:473)
>         at org.junit.Assert.assertArrayEquals(Assert.java:294)
>         at org.apache.hadoop.io.compress.CompressDecompressTester$CompressionTestStrategy$2.assertCompression(CompressDecompressTester.java:325)
>         at org.apache.hadoop.io.compress.CompressDecompressTester.test(CompressDecompressTester.java:135)
>         at org.apache.hadoop.io.compress.TestCompressorDecompressor.testCompressorDecompressor(TestCompressorDecompressor.java:58)
> .......
> .......
> .....



--
This message was sent by Atlassian JIRA
(v6.2#6252)