You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2019/12/21 15:35:47 UTC
[spark] branch branch-2.4 updated:
[SPARK-29918][SQL][FOLLOWUP][TEST] Fix arrayOffset in
`RecordBinaryComparatorSuite`
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 871aacc [SPARK-29918][SQL][FOLLOWUP][TEST] Fix arrayOffset in `RecordBinaryComparatorSuite`
871aacc is described below
commit 871aacc11a4ffe36b89395171c04ee1e1a2a4dda
Author: Xingbo Jiang <xi...@databricks.com>
AuthorDate: Thu Dec 19 17:01:40 2019 +0800
[SPARK-29918][SQL][FOLLOWUP][TEST] Fix arrayOffset in `RecordBinaryComparatorSuite`
### What changes were proposed in this pull request?
As mentioned in https://github.com/apache/spark/pull/26548#pullrequestreview-334345333, some test cases in `RecordBinaryComparatorSuite` use a fixed arrayOffset when writing to long arrays, this could lead to weird stuff including crashing with a SIGSEGV.
This PR fix the problem by computing the arrayOffset based on `Platform.LONG_ARRAY_OFFSET`.
### How was this patch tested?
Tested locally. Previously, when we try to add `System.gc()` between write into long array and compare by RecordBinaryComparator, there is a chance to hit JVM crash with SIGSEGV like:
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007efc66970bcb, pid=11831, tid=0x00007efc0f9f9700
#
# JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-8u222-b10-1ubuntu1~16.04.1-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x5fbbcb]
#
# Core dump written. Default location: /home/jenkins/workspace/sql/core/core or core.11831
#
# An error report file with more information is saved as:
# /home/jenkins/workspace/sql/core/hs_err_pid11831.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
```
After the fix those test cases didn't crash the JVM anymore.
Closes #26939 from jiangxb1987/rbc.
Authored-by: Xingbo Jiang <xi...@databricks.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.../apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/sql/core/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java b/sql/core/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java
index 9664638e..c832d07 100644
--- a/sql/core/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java
+++ b/sql/core/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java
@@ -323,7 +323,7 @@ public class RecordBinaryComparatorSuite {
@Test
public void testCompareLongsAsLittleEndian() {
- long arrayOffset = 12;
+ long arrayOffset = Platform.LONG_ARRAY_OFFSET + 4;
long[] arr1 = new long[2];
Platform.putLong(arr1, arrayOffset, 0x0100000000000000L);
@@ -345,7 +345,7 @@ public class RecordBinaryComparatorSuite {
@Test
public void testCompareLongsAsUnsigned() {
- long arrayOffset = 12;
+ long arrayOffset = Platform.LONG_ARRAY_OFFSET + 4;
long[] arr1 = new long[2];
Platform.putLong(arr1, arrayOffset + 4, 0xa000000000000000L);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org