You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/06/09 15:25:06 UTC
[GitHub] [spark] kiszk commented on issue #24788: [SPARK-26985][CORE] Fix
"access only some column of the all of columns " for big endian architecture
kiszk commented on issue #24788: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture
URL: https://github.com/apache/spark/pull/24788#issuecomment-500220841
I double-checked some of tests. I confirmed this PR fixes existing failures on big-endian platform. Thank you very much.
**Before**
```
$ uname -a
Linux ppc64be.novalocal 4.5.7-300.fc24.ppc64 #1 SMP Fri Jun 10 20:29:32 UTC 2016 ppc64 ppc64 ppc64 GNU/Linux
$ build/sbt "sql/testOnly *.ColumnVectorSuite"
...
[info] ColumnVectorSuite:
[info] - boolean (194 milliseconds)
[info] - byte (4 milliseconds)
[info] - short (4 milliseconds)
[info] - int (4 milliseconds)
[info] - date (4 milliseconds)
[info] - long (4 milliseconds)
[info] - timestamp (5 milliseconds)
[info] - float (4 milliseconds)
[info] - double (4 milliseconds)
[info] - string (14 milliseconds)
[info] - binary (8 milliseconds)
[info] - mutable ColumnarRow (4 milliseconds)
[info] - array (1 millisecond)
[info] - struct (1 millisecond)
[info] - [SPARK-22092] off-heap column vector reallocation corrupts array data (3 milliseconds)
[info] - [SPARK-22092] off-heap column vector reallocation corrupts struct nullability (3 milliseconds)
[info] - CachedBatch boolean Apis (46 milliseconds)
[info] - CachedBatch byte Apis (7 milliseconds)
[info] - CachedBatch short Apis (8 milliseconds)
[info] - CachedBatch int Apis (9 milliseconds)
[info] - CachedBatch long Apis (8 milliseconds)
[info] - CachedBatch float Apis *** FAILED *** (30 milliseconds)
[info] 4.6006E-41 did not equal 1.0 (ColumnVectorSuite.scala:424)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
[info] at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$71(ColumnVectorSuite.scala:424)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$71$adapted(ColumnVectorSuite.scala:422)
[info] at scala.collection.immutable.Range.foreach(Range.scala:158)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$70(ColumnVectorSuite.scala:422)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$70$adapted(ColumnVectorSuite.scala:417)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.withVector(ColumnVectorSuite.scala:34)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.withVectors(ColumnVectorSuite.scala:41)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$68(ColumnVectorSuite.scala:417)
[info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
[info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
[info] at org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
[info] at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
[info] at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
[info] at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
[info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
[info] at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
[info] at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
[info] at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
[info] at org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
[info] at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
[info] at scala.collection.immutable.List.foreach(List.scala:392)
[info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
[info] at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
[info] at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
[info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
[info] at org.scalatest.Suite.run(Suite.scala:1147)
[info] at org.scalatest.Suite.run$(Suite.scala:1129)
[info] at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
[info] at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
[info] at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
[info] at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
[info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:296)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:286)
[info] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[info] at java.lang.Thread.run(Thread.java:748)
[info] - CachedBatch double Apis *** FAILED *** (11 milliseconds)
[info] 3.03865E-319 did not equal 1.0 (ColumnVectorSuite.scala:448)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
[info] at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$75(ColumnVectorSuite.scala:448)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$75$adapted(ColumnVectorSuite.scala:446)
[info] at scala.collection.immutable.Range.foreach(Range.scala:158)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$74(ColumnVectorSuite.scala:446)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$74$adapted(ColumnVectorSuite.scala:441)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.withVector(ColumnVectorSuite.scala:34)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.withVectors(ColumnVectorSuite.scala:41)
[info] at org.apache.spark.sql.execution.vectorized.ColumnVectorSuite.$anonfun$new$72(ColumnVectorSuite.scala:441)
[info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
[info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
[info] at org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
[info] at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
[info] at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
[info] at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
[info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
[info] at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
[info] at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
[info] at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
[info] at org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
[info] at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
[info] at scala.collection.immutable.List.foreach(List.scala:392)
[info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
[info] at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
[info] at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
[info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
[info] at org.scalatest.Suite.run(Suite.scala:1147)
[info] at org.scalatest.Suite.run$(Suite.scala:1129)
[info] at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
[info] at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
[info] at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
[info] at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
[info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:296)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:286)
[info] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[info] at java.lang.Thread.run(Thread.java:748)
[info] ScalaTest
[info] Run completed in 6 seconds, 911 milliseconds.
[info] Total number of tests run: 23
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 21, failed 2, canceled 0, ignored 0, pending 0
[info] *** 2 TESTS FAILED ***
[error] Failed: Total 23, Failed 2, Errors 0, Passed 21
[error] Failed tests:
[error] org.apache.spark.sql.execution.vectorized.ColumnVectorSuite
[error] (sql/test:testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 700 s, completed Jun 9, 2019 11:23:46 PM
```
**After**
```
$ uname -a
Linux ppc64be.novalocal 4.5.7-300.fc24.ppc64 #1 SMP Fri Jun 10 20:29:32 UTC 2016 ppc64 ppc64 ppc64 GNU/Linux
$ build/sbt "sql/testOnly *.ColumnVectorSuite"
...
[info] ColumnVectorSuite:
[info] - boolean (187 milliseconds)
[info] - byte (6 milliseconds)
[info] - short (6 milliseconds)
[info] - int (5 milliseconds)
[info] - date (4 milliseconds)
[info] - long (4 milliseconds)
[info] - timestamp (4 milliseconds)
[info] - float (4 milliseconds)
[info] - double (4 milliseconds)
[info] - string (13 milliseconds)
[info] - binary (8 milliseconds)
[info] - mutable ColumnarRow (5 milliseconds)
[info] - array (1 millisecond)
[info] - struct (2 milliseconds)
[info] - [SPARK-22092] off-heap column vector reallocation corrupts array data (4 milliseconds)
[info] - [SPARK-22092] off-heap column vector reallocation corrupts struct nullability (3 milliseconds)
[info] - CachedBatch boolean Apis (45 milliseconds)
[info] - CachedBatch byte Apis (7 milliseconds)
[info] - CachedBatch short Apis (7 milliseconds)
[info] - CachedBatch int Apis (9 milliseconds)
[info] - CachedBatch long Apis (8 milliseconds)
[info] - CachedBatch float Apis (8 milliseconds)
[info] - CachedBatch double Apis (7 milliseconds)
[info] ScalaTest
[info] Run completed in 6 seconds, 733 milliseconds.
[info] Total number of tests run: 23
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 23, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 23, Failed 0, Errors 0, Passed 23
[success] Total time: 38 s, completed Jun 9, 2019 11:52:54 PM
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org