You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Anuja Jakhade (JIRA)" <ji...@apache.org> on 2019/03/19 10:13:00 UTC
[jira] [Comment Edited] (SPARK-26985) Test "access only some column
of the all of columns " fails on big endian
[ https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795931#comment-16795931 ]
Anuja Jakhade edited comment on SPARK-26985 at 3/19/19 10:12 AM:
-----------------------------------------------------------------
Hi [~srowen], [~hyukjin.kwon]
I have observed that after changing the ByteOrder in "_*OnHeapColumnVector.java*_" to *ByteOrder.BIG_ENDIAN* the tests passes. Because the data is read properly. However in that case some tests of Paraquet Module fails. e.x: *ParquetIOSuite.*
Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN even when the bigEndianPlatform is true.
Because in that case the fix doesn't work on all the test cases and the behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment each other.
was (Author: anuja):
Hi [~srowen], [~hyukjin.kwon]
I have observed that after changing the ByteOrder in "_*OnHeapColumnVector.java*_" to *ByteOrder.LITTLE_ENDIAN* the tests passes. Because the data is read properly. However in that case some tests of Paraquet Module fails. e.x: *ParquetIOSuite.*
Is there any specific reason why we are using ByteOrder format as LITTLE_ENDIAN even when the bigEndianPlatform is true.
Because in that case the fix doesn't work on all the test cases and the behavior of *ParquetIOSuite and DataFrameTungsten/InMemoryColumnarQuerySuite* compliment each other.
> Test "access only some column of the all of columns " fails on big endian
> -------------------------------------------------------------------------
>
> Key: SPARK-26985
> URL: https://issues.apache.org/jira/browse/SPARK-26985
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.2
> Environment: Linux Ubuntu 16.04
> openjdk version "1.8.0_202"
> OpenJDK Runtime Environment (build 1.8.0_202-b08)
> Eclipse OpenJ9 VM (build openj9-0.12.1, JRE 1.8.0 64-Bit Compressed References 20190205_218 (JIT enabled, AOT enabled)
> OpenJ9 - 90dd8cb40
> OMR - d2f4534b
> JCL - d002501a90 based on jdk8u202-b08)
>
> Reporter: Anuja Jakhade
> Priority: Major
> Labels: BigEndian
> Attachments: DataFrameTungstenSuite.txt, InMemoryColumnarQuerySuite.txt, access only some column of the all of columns.txt
>
>
> While running tests on Apache Spark v2.3.2 with AdoptJDK on big endian, I am observing test failures for 2 Suites of Project SQL.
> 1. InMemoryColumnarQuerySuite
> 2. DataFrameTungstenSuite
> In both the cases test "access only some column of the all of columns" fails due to mismatch in the final assert.
> Observed that the data obtained after df.cache() is causing the error. Please find attached the log with the details.
> cache() works perfectly fine if double and float values are not in picture.
> Inside test !!!!!!- access only some column of the all of columns *** FAILED ***
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org