[jira] [Commented] (ARROW-3476) [Java] mvn test in memory fails on a big-endian platform

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651642#comment-16651642 ] 

Wes McKinney commented on ARROW-3476:
-------------------------------------

[~kiszk] there is nothing structurally preventing the Arrow columnar format from being used on big-endian platforms, it is an implementation and testing question. If IBM could make some hardware available for testing and development, it would be a benefit to the project. 

The columnar memory format itself uses little endian integer representation, so byte swapping would be necessary in many places. I'm interested to hear how big endian support could be introduced in Java and C++ without introducing performance regressions on little endian systems (particular in Java).

> [Java] mvn test in memory fails on a big-endian platform
> --------------------------------------------------------
>
>                 Key: ARROW-3476
>                 URL: https://issues.apache.org/jira/browse/ARROW-3476
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Java
>            Reporter: Kazuaki Ishizaki
>            Priority: Major
>
> Apache Arrow is becoming commonplace to exchange data among important emerging analytics frameworks such as Pandas, Numpy, and Spark.
> [IBM Z|https://en.wikipedia.org/wiki/IBM_Z] is one of platforms to process critical transactions such as bank or credit card. Users of IBM Z want to extract insights from these transactions using the emerging analytics systems on IBM Z Linux. These analytics pipelines can be also fast and effective on IBM Z Linux by using Apache Arrow on memory.
> From the technical perspective, since IBM Z Linux uses big-endian data format, it is not possible to use Apache Arrow in this pipeline. If Apache Arrow could support big-endian, the use case would be expanded.
> When I ran test case of Apache arrow on a big-endian platform (ppc64be), {{mvn test}} in memory causes a failure due to an assertion.
> In {{TestEndianess.testLittleEndian}} test suite, the assertion occurs during an allocation of a {{RootAllocator}} class.
> {code}
> $ uname -a
> Linux ppc64be.novalocal 4.5.7-300.fc24.ppc64 #1 SMP Fri Jun 10 20:29:32 UTC 2016 ppc64 ppc64 ppc64 GNU/Linux
> $ arch  
> ppc64
> $ cd java/memory
> $ mvn test
> [INFO] Scanning for projects...
> [INFO]                                                                         
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Arrow Memory 0.12.0-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> [INFO] 
> ...
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 s - in org.apache.arrow.memory.TestAccountant
> [INFO] Running org.apache.arrow.memory.TestLowCostIdentityHashMap
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 s - in org.apache.arrow.memory.TestLowCostIdentityHashMap
> [INFO] Running org.apache.arrow.memory.TestBaseAllocator
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.746 s <<< FAILURE! - in org.apache.arrow.memory.TestEndianess
> [ERROR] testLittleEndian(org.apache.arrow.memory.TestEndianess)  Time elapsed: 0.313 s  <<< ERROR!
> java.lang.ExceptionInInitializerError
> 	at org.apache.arrow.memory.TestEndianess.testLittleEndian(TestEndianess.java:31)
> Caused by: java.lang.IllegalStateException: Arrow only runs on LittleEndian systems.
> 	at org.apache.arrow.memory.TestEndianess.testLittleEndian(TestEndianess.java:31)
> [ERROR] Tests run: 22, Failures: 0, Errors: 21, Skipped: 1, Time elapsed: 0.055 s <<< FAILURE! - in org.apache.arrow.memory.TestBaseAllocator
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)