You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/07 02:52:46 UTC

[GitHub] [spark] tinhto-000 opened a new pull request #29383: [SPARK-31703] Parquet RLE float/double are read incorrectly on big endian platforms

tinhto-000 opened a new pull request #29383:
URL: https://github.com/apache/spark/pull/29383


   ### What changes were proposed in this pull request?
   This PR fixes the issue introduced during SPARK-26985.
   
   SPARK-26985 changes the `putDoubles()` and `putFloats()` methods to respect the platform's endian-ness.  However, that causes the RLE paths in VectorizedRleValuesReader.java to read the RLE entries in parquet as BIG_ENDIAN on big endian platforms, even though parquet data is always in little endian format.
   
   The comments in `WriteableColumnVector.java` say those methods are used for "ieee formatted doubles in platform native endian" (or floats), but since the data in parquet is always in little endian format, use of those methods appears to be inappropriate.
   
   To demonstrate the problem with spark-shell:
   
   ```
    import org.apache.spark._
   import org.apache.spark.sql._
   import org.apache.spark.sql.types._
       
   var data = Seq(
     (1.0, 0.1),
     (2.0, 0.2),
     (0.3, 3.0),
     (4.0, 4.0),
     (5.0, 5.0))
     
   var df = spark.createDataFrame(data).write.mode(SaveMode.Overwrite).parquet("/tmp/data.parquet2")
   var df2 = spark.read.parquet("/tmp/data.parquet2")
   df2.show()
   ```
   
   result:
   
   ```
   +--------------------+--------------------+                                     
   |                  _1|                  _2|
   +--------------------+--------------------+
   |           3.16E-322|-1.54234871366845...|
   |         2.0553E-320|         2.0553E-320|
   |          2.561E-320|          2.561E-320|
   |4.66726145843124E-62|         1.0435E-320|
   |        3.03865E-319|-1.54234871366757...|
   +--------------------+--------------------+
   ```
   
   Also tests in ParquetIOSuite that involve float/double data would fail, e.g., 
   
   - basic data types (without binary)
   - read raw Parquet file
   
   /examples/src/main/python/mllib/isotonic_regression_example.py would fail as well.
   
   Purposed code change is to add `putDoublesLittleEndian()` and `putFloatsLittleEndian()` methods for parquet to invoke, just like the existing `putIntsLittleEndian()` and `putLongsLittleEndian()`.  On little endian platforms they would call `putDoubles()` and `putFloats()`, on big endian they would read the entries as little endian like pre-SPARK-26985.
   
   ### Why are the changes needed?
   RLE float/double data in parquet files will not be read back correctly on big endian platforms.
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   All unit tests (mvn test) were ran and OK.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670994867


   cc @ScrapCodes since this is a correctness issue.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan closed pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29383:
URL: https://github.com/apache/spark/pull/29383


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670480765


   **[Test build #127203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127203/testReport)** for PR 29383 at commit [`2261c09`](https://github.com/apache/spark/commit/2261c09e5fb6090372022517de67ad02f2c3dad6).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672275557


   **[Test build #127336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127336/testReport)** for PR 29383 at commit [`f160da0`](https://github.com/apache/spark/commit/f160da0035d8b1df08b75832f5871f7afc5cb839).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672070547


   **[Test build #127336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127336/testReport)** for PR 29383 at commit [`f160da0`](https://github.com/apache/spark/commit/f160da0035d8b1df08b75832f5871f7afc5cb839).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-671285628


   Looks fine, too.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670481141






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] tinhto-000 commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
tinhto-000 commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672072900


   @kiszk thanks for the great suggestion and point out.  Turned out ColumnarBatchSuite.scala failed on big endian as well.
   
   The "Double APIs" and "Float APIs" are using `putFloats()` and `putDoubles()` on little endian stream they explicitly created , and those failed on big endian.
   
   I modified those 2 tests and added 2 [SPARK-31703] tests, and the tests look fine on both x86 and z/OS (s390) now.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670607214






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670590810


   cc @vibhutisawant since he is the original author.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29383:
URL: https://github.com/apache/spark/pull/29383#discussion_r467178364



##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java
##########
@@ -425,6 +425,19 @@ public void putFloats(int rowId, int count, byte[] src, int srcIndex) {
     }
   }
 
+  @Override
+  public void putFloatsLittleEndian(int rowId, int count, byte[] src, int srcIndex) {
+    if (!bigEndianPlatform) {

Review comment:
       Just leave it as-is for consistency, yeah. OK I see we have lots of "as little endian" variants so maybe this is the best approach for symmetry.
   
   I dont' really know enough about this part to validate the change but it seems plausible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670594264


   Seems reasonable - is there any case where we would need putFloats separately from putFloatsLittleEndian? like would you ever legitimately not care about endianness of the serialization? (may be a dumb question)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670481141






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] tinhto-000 commented on a change in pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
tinhto-000 commented on a change in pull request #29383:
URL: https://github.com/apache/spark/pull/29383#discussion_r467154975



##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java
##########
@@ -425,6 +425,19 @@ public void putFloats(int rowId, int count, byte[] src, int srcIndex) {
     }
   }
 
+  @Override
+  public void putFloatsLittleEndian(int rowId, int count, byte[] src, int srcIndex) {
+    if (!bigEndianPlatform) {

Review comment:
       agreed, it definitely will look better.  will change in next commit.  thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29383: [SPARK-31703] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670293331


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672071427






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29383:
URL: https://github.com/apache/spark/pull/29383#discussion_r467136633



##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java
##########
@@ -425,6 +425,19 @@ public void putFloats(int rowId, int count, byte[] src, int srcIndex) {
     }
   }
 
+  @Override
+  public void putFloatsLittleEndian(int rowId, int count, byte[] src, int srcIndex) {
+    if (!bigEndianPlatform) {

Review comment:
       Really doesn't matter, but you could flip the comparison here to avoid the negation




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] kiszk commented on a change in pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #29383:
URL: https://github.com/apache/spark/pull/29383#discussion_r469451763



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala
##########
@@ -447,13 +447,6 @@ class ColumnarBatchSuite extends SparkFunSuite {
       Platform.putFloat(buffer, Platform.BYTE_ARRAY_OFFSET, 2.234f)
       Platform.putFloat(buffer, Platform.BYTE_ARRAY_OFFSET + 4, 1.123f)
 
-      if (ByteOrder.nativeOrder().equals(ByteOrder.BIG_ENDIAN)) {

Review comment:
       Thank you. This code did not make sense due to [the old implementation](https://github.com/tinhto-000/spark/commit/8a1ce4899fb9f751dedaaa34ea654dfbc8330852)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #29383: [SPARK-31703] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670479830


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] tinhto-000 commented on a change in pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
tinhto-000 commented on a change in pull request #29383:
URL: https://github.com/apache/spark/pull/29383#discussion_r467159716



##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java
##########
@@ -425,6 +425,19 @@ public void putFloats(int rowId, int count, byte[] src, int srcIndex) {
     }
   }
 
+  @Override
+  public void putFloatsLittleEndian(int rowId, int count, byte[] src, int srcIndex) {
+    if (!bigEndianPlatform) {

Review comment:
       Actually now I look at the existing `putIntsLittleEndian()` and `putLongsLittleEndian()` methods and they do `if (!bigEndianPlatform)` as well (and thus how I copied and pasted from those).
   
   @srowen Do you think we can leave the negation alone for uniformity, or go ahead and change?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] tinhto-000 commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
tinhto-000 commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670612143


   > Seems reasonable - is there any case where we would need putFloats separately from putFloatsLittleEndian? like would you ever legitimately not care about endianness of the serialization? (may be a dumb question)
   
   It's a good question.  It *seems* like column compression drives putFloats() and I'm *guessing* that's the reason for the original SPARK-26985.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670607214






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] kiszk commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-671011567


   Good catch. This change looks good.   
   Could you please add test cases to [ColumnarBatchSuite.scala](https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala) like `putIntsLittleEndian()`?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] tinhto-000 edited a comment on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
tinhto-000 edited a comment on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672072900






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-671209885


   LGTM. +1 for adding tests


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670480765


   **[Test build #127203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127203/testReport)** for PR 29383 at commit [`2261c09`](https://github.com/apache/spark/commit/2261c09e5fb6090372022517de67ad02f2c3dad6).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672276640






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672640923


   thanks, merging to master/3.0!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670489216


   also cc @ScrapCodes and @zhengruifeng because this bug is realted to branch-3.0/branch-2.4 and they are the release managers. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670481822


   cc: @rednaxelafx @kiszk @srowen 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29383: [SPARK-31703] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670293612


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] tinhto-000 commented on a change in pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
tinhto-000 commented on a change in pull request #29383:
URL: https://github.com/apache/spark/pull/29383#discussion_r467159716



##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java
##########
@@ -425,6 +425,19 @@ public void putFloats(int rowId, int count, byte[] src, int srcIndex) {
     }
   }
 
+  @Override
+  public void putFloatsLittleEndian(int rowId, int count, byte[] src, int srcIndex) {
+    if (!bigEndianPlatform) {

Review comment:
       Actually now I look at the existing `putDoublesLittleEndian()` and `putFloatsLittleEndian()` methods and they do `if (!bigEndianPlatform)` as well (and thus how I copied and pasted from those).
   
   @srowen Do you think we can leave the negation alone for uniformity, or go ahead and change?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672276640






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672071427






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29383: [SPARK-31703] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670293331


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670293612


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-672070547


   **[Test build #127336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127336/testReport)** for PR 29383 at commit [`f160da0`](https://github.com/apache/spark/commit/f160da0035d8b1df08b75832f5871f7afc5cb839).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670995006


   I raised the priority of SPARK-31703 to `Blocker` with `Target Version` 2.4.7 and 3.0.1.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29383: [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29383:
URL: https://github.com/apache/spark/pull/29383#issuecomment-670606333


   **[Test build #127203 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127203/testReport)** for PR 29383 at commit [`2261c09`](https://github.com/apache/spark/commit/2261c09e5fb6090372022517de67ad02f2c3dad6).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org