[spark] branch master updated: [SPARK-39519][SPARK-39387][SQL][TESTS] Ignore flaky test `: BytesColumnVector should not throw RuntimeException due to overflow`

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 51194d883c6 [SPARK-39519][SPARK-39387][SQL][TESTS] Ignore flaky test `: BytesColumnVector should not throw RuntimeException due to overflow`
51194d883c6 is described below

commit 51194d883c63187423b9069cf1d2f040207566f6
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Fri Jun 24 11:58:28 2022 +0900

    [SPARK-39519][SPARK-39387][SQL][TESTS] Ignore flaky test `: BytesColumnVector should not throw RuntimeException due to overflow`
    
    ### What changes were proposed in this pull request?
    This pr aims to ignore flaky test `SPARK-39387: BytesColumnVector should not throw RuntimeException due to overflow` in `OrcQuerySuite` to ensure test stability when using Java 11.
    
    ### Why are the changes needed?
    "SPARK-39387: BytesColumnVector should not throw RuntimeException due to overflow" will prepare a large amount of test data (100MB * 22) and involve the allocation of multiple large arrays(`sharedBuffer` in `BytesColumnVector#allocateBuffer` method), the biggest is 1GB(From the description of https://github.com/apache/spark/pull/36772, it is expected). So when using Java 11 to test whole `sql` module, this suite often triggers OOM.
    
    This pr aims to ensure the stability of the test when using Java 11.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Pass GA
    
    Closes #36954 from LuciferYang/NewRatio.
    
    Authored-by: yangjie01 <ya...@baidu.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 .../apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala    | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
index 275f20a08e6..2152cbd263a 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
@@ -833,7 +833,9 @@ abstract class OrcQuerySuite extends OrcQueryTest with SharedSparkSession {
     }
   }
 
-  test("SPARK-39387: BytesColumnVector should not throw RuntimeException due to overflow") {
+  // SPARK-39519: Ignore this case because it requires more than 4g heap memory to ensure test
+  // stability when use Java 11. Should test it manually when upgrading `hive-storage-api`
+  ignore("SPARK-39387: BytesColumnVector should not throw RuntimeException due to overflow") {
     withTempPath { dir =>
       val path = dir.getCanonicalPath
       val df = spark.range(1, 22, 1, 1).map { _ =>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org