You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Pete Robbins (JIRA)" <ji...@apache.org> on 2016/04/14 10:28:25 UTC
[jira] [Commented] (SPARK-13745) Support columnar in memory
representation on Big Endian platforms
[ https://issues.apache.org/jira/browse/SPARK-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240802#comment-15240802 ]
Pete Robbins commented on SPARK-13745:
--------------------------------------
I have started working on this. The changes required are in:
org.apache.spark.sql.execution.datasources.parquet.VectorizedPlainValuesReader
org.apache.spark.sql.execution.vectorized.OnHeapColumnVector
org.apache.spark.sql.execution.vectorized.OffHeapColumnVector
when interpreting ints, longs, doubles and floats from byte[] which are in Little Endian
> Support columnar in memory representation on Big Endian platforms
> -----------------------------------------------------------------
>
> Key: SPARK-13745
> URL: https://issues.apache.org/jira/browse/SPARK-13745
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Tim Preece
> Labels: big-endian
>
> SPARK-12785 introduced a columnar in memory representation.
> Currently this feature is explicitly only supported on Little Endian platorms. On Big Endian platforms the following exception is thrown:
> "org.apache.commons.lang.NotImplementedException: Only little endian is supported."
> This JIRA should be used to extend support to Big Endian architectures, and decide whether the "in memory" columnar format should be consistent with parquet format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org