You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/30 04:09:48 UTC

[GitHub] [spark] LuciferYang opened a new pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

LuciferYang opened a new pull request #34153:
URL: https://github.com/apache/spark/pull/34153


   ### What changes were proposed in this pull request?
   To be added ...
   
   
   
   
   ### Why are the changes needed?
   Pass Spark UTs with JDK 17
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   To be added ...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939751470


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48547/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941875685


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48650/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941928030


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144172/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939646939


   **[Test build #144065 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144065/testReport)** for PR 34153 at commit [`4aa3643`](https://github.com/apache/spark/commit/4aa36437d07c152546975c6e32278fe2b605759f).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-943335668


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48738/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946388247


   **[Test build #144396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144396/testReport)** for PR 34153 at commit [`c7265c9`](https://github.com/apache/spark/commit/c7265c9168cb4669e9465f88e42a284f8a35dad0).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946454763






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940693876


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48598/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939786068


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48547/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939783494


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48547/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939846149


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144068/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940640077


   **[Test build #144121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144121/testReport)** for PR 34153 at commit [`fdd912e`](https://github.com/apache/spark/commit/fdd912ed3f2114f57dcdca981591b87ec918a24b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930775606


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48257/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941855485


   **[Test build #144172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144172/testReport)** for PR 34153 at commit [`ee8b1b8`](https://github.com/apache/spark/commit/ee8b1b8d070d168e93ae21319b2b4d9737416f74).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930776697


   **[Test build #143747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143747/testReport)** for PR 34153 at commit [`70fc3bf`](https://github.com/apache/spark/commit/70fc3bf618c909baf2eb65ab25b1aaf4376b0fea).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719128295



##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql-jdk17.out
##########
@@ -0,0 +1,360 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 42
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text string') AS true
+-- !query schema
+struct<true:boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text strin') AS `false`
+-- !query schema
+struct<false:boolean>
+-- !query output
+false
+
+
+-- !query
+CREATE TABLE TEXT_TBL (f1 string) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('doh!')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('hi de ho neighbor')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT '' AS two, * FROM TEXT_TBL
+-- !query schema
+struct<two:string,f1:string>
+-- !query output
+	doh!
+	hi de ho neighbor
+
+
+-- !query
+select length(42)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'length(42)' due to data type mismatch: argument 1 requires (string or binary) type, however, '42' is of int type.; line 1 pos 7
+
+
+-- !query
+select string('four: ') || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(CAST('four: ' AS STRING), 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 'four: ' || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat('four: ', 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 3 || 4.0
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(3, 4.0BD)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, decimal(2,1)]; line 1 pos 7
+
+
+-- !query
+/*
+ * various string functions
+ */
+select concat('one')
+-- !query schema
+struct<concat(one):string>
+-- !query output
+one
+
+
+-- !query
+select concat(1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, int, int, string, boolean, boolean, date]; line 1 pos 7
+
+
+-- !query
+select concat_ws('#','one')
+-- !query schema
+struct<concat_ws(#, one):string>
+-- !query output
+one
+
+
+-- !query
+select concat_ws('#',1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('#', 1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '1' is of int type. argument 3 requires (array<string> or string) type, however, '2' is of int type. argument 4 requires (array<string> or string) type, however, '3' is of int type. argument 6 requires (array<string> or string) type, however, 'true' is of boolean type. argument 7 requires (array<string> or string) type, however, 'false' is of boolean type. argument 8 requires (array<string> or string) type, however, 'to_date('20100309', 'yyyyMMdd')' is of date type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(',',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(',', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws('',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(NULL,10,20,null,30) is null
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(CAST(NULL AS STRING), 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select reverse('abcde')
+-- !query schema
+struct<reverse(abcde):string>
+-- !query output
+edcba
+
+
+-- !query
+select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'substring('ahoj', 1, t.i)' due to data type mismatch: argument 3 requires int type, however, 't.i' is of bigint type.; line 1 pos 10
+
+
+-- !query
+/*
+ * format
+ */
+select format_string(NULL)
+-- !query schema
+struct<format_string(NULL):string>
+-- !query output
+NULL
+
+
+-- !query
+select format_string('Hello')
+-- !query schema
+struct<format_string(Hello):string>
+-- !query output
+Hello
+
+
+-- !query
+select format_string('Hello %s', 'World')
+-- !query schema
+struct<format_string(Hello %s, World):string>
+-- !query output
+Hello World
+
+
+-- !query
+select format_string('Hello %%')
+-- !query schema
+struct<format_string(Hello %%):string>
+-- !query output
+Hello %
+
+
+-- !query
+select format_string('Hello %%%%')
+-- !query schema
+struct<format_string(Hello %%%%):string>
+-- !query output
+Hello %%
+
+
+-- !query
+select format_string('Hello %s %s', 'World')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %s')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %x', 20)
+-- !query schema
+struct<format_string(Hello %x, 20):string>
+-- !query output
+Hello 14
+
+
+-- !query
+select format_string('%1$s %3$s', 1, 2, 3)
+-- !query schema
+struct<format_string(%1$s %3$s, 1, 2, 3):string>
+-- !query output
+1 3
+
+
+-- !query
+select format_string('%1$s %12$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<format_string(%1$s %12$s, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12):string>
+-- !query output
+1 12
+
+
+-- !query
+select format_string('%1$s %4$s', 1, 2, 3)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%4$s'
+
+
+-- !query
+select format_string('%1$s %13$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%13$s'
+
+
+-- !query
+select format_string('%0$s', 'Hello')
+-- !query schema
+struct<>
+-- !query output
+java.util.IllegalFormatArgumentIndexException
+Illegal format argument index = 0
+

Review comment:
       ```
   --PostgreSQL throw ERROR:  format specifies argument 0, but arguments are numbered from 1
   select format_string('%0$s', 'Hello');
   ```
   The SQL in `postgreSQL/text.sql` has this different behavior between Java8 and Java17, but it seems that the behavior of Java 17 is expected, so do Independent verification here.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930953526


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48265/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r725795555



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSparkSubmitSuite.scala
##########
@@ -40,15 +40,26 @@ class WholeStageCodegenSparkSubmitSuite extends SparkSubmitTestUtils
     val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
 
     // HotSpot JVM specific: Set up a local cluster with the driver/executor using mismatched
-    // settings of UseCompressedOops JVM option.
+    // settings of UseCompressedClassPointers JVM option.
     val argsForSparkSubmit = Seq(
       "--class", WholeStageCodegenSparkSubmitSuite.getClass.getName.stripSuffix("$"),
       "--master", "local-cluster[1,1,1024]",
       "--driver-memory", "1g",
       "--conf", "spark.ui.enabled=false",
       "--conf", "spark.master.rest.enabled=false",
-      "--conf", "spark.driver.extraJavaOptions=-XX:-UseCompressedOops",
-      "--conf", "spark.executor.extraJavaOptions=-XX:+UseCompressedOops",
+      // SPARK-36796: The results of `Platform.BYTE_ARRAY_OFFSET` using different Java versions

Review comment:
       cc @dongjoon-hyun @srowen What do you think about this case? In Java 8, `UseCompressedOops` and `UseCompressedClassPointers` are both passed. But In Java 17, `UseCompressedClassPointers` should be used.  Should make this a separate PR? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940694569


   **[Test build #144126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144126/testReport)** for PR 34153 at commit [`2e87bde`](https://github.com/apache/spark/commit/2e87bdeff7500bbf50d18efa25b02205b48a0dff).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939850657


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144069/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939923508


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48553/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940700519


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144121/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940163729


   **[Test build #144090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144090/testReport)** for PR 34153 at commit [`0e4bba8`](https://github.com/apache/spark/commit/0e4bba8e37a1af76da8f21a49cf026abee6d5b4b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r725798950



##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql-jdk17.out
##########
@@ -0,0 +1,360 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 42
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text string') AS true
+-- !query schema
+struct<true:boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text strin') AS `false`
+-- !query schema
+struct<false:boolean>
+-- !query output
+false
+
+
+-- !query
+CREATE TABLE TEXT_TBL (f1 string) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('doh!')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('hi de ho neighbor')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT '' AS two, * FROM TEXT_TBL
+-- !query schema
+struct<two:string,f1:string>
+-- !query output
+	doh!
+	hi de ho neighbor
+
+
+-- !query
+select length(42)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'length(42)' due to data type mismatch: argument 1 requires (string or binary) type, however, '42' is of int type.; line 1 pos 7
+
+
+-- !query
+select string('four: ') || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(CAST('four: ' AS STRING), 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 'four: ' || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat('four: ', 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 3 || 4.0
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(3, 4.0BD)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, decimal(2,1)]; line 1 pos 7
+
+
+-- !query
+/*
+ * various string functions
+ */
+select concat('one')
+-- !query schema
+struct<concat(one):string>
+-- !query output
+one
+
+
+-- !query
+select concat(1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, int, int, string, boolean, boolean, date]; line 1 pos 7
+
+
+-- !query
+select concat_ws('#','one')
+-- !query schema
+struct<concat_ws(#, one):string>
+-- !query output
+one
+
+
+-- !query
+select concat_ws('#',1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('#', 1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '1' is of int type. argument 3 requires (array<string> or string) type, however, '2' is of int type. argument 4 requires (array<string> or string) type, however, '3' is of int type. argument 6 requires (array<string> or string) type, however, 'true' is of boolean type. argument 7 requires (array<string> or string) type, however, 'false' is of boolean type. argument 8 requires (array<string> or string) type, however, 'to_date('20100309', 'yyyyMMdd')' is of date type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(',',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(',', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws('',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(NULL,10,20,null,30) is null
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(CAST(NULL AS STRING), 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select reverse('abcde')
+-- !query schema
+struct<reverse(abcde):string>
+-- !query output
+edcba
+
+
+-- !query
+select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'substring('ahoj', 1, t.i)' due to data type mismatch: argument 3 requires int type, however, 't.i' is of bigint type.; line 1 pos 10
+
+
+-- !query
+/*
+ * format
+ */
+select format_string(NULL)
+-- !query schema
+struct<format_string(NULL):string>
+-- !query output
+NULL
+
+
+-- !query
+select format_string('Hello')
+-- !query schema
+struct<format_string(Hello):string>
+-- !query output
+Hello
+
+
+-- !query
+select format_string('Hello %s', 'World')
+-- !query schema
+struct<format_string(Hello %s, World):string>
+-- !query output
+Hello World
+
+
+-- !query
+select format_string('Hello %%')
+-- !query schema
+struct<format_string(Hello %%):string>
+-- !query output
+Hello %
+
+
+-- !query
+select format_string('Hello %%%%')
+-- !query schema
+struct<format_string(Hello %%%%):string>
+-- !query output
+Hello %%
+
+
+-- !query
+select format_string('Hello %s %s', 'World')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %s')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %x', 20)
+-- !query schema
+struct<format_string(Hello %x, 20):string>
+-- !query output
+Hello 14
+
+
+-- !query
+select format_string('%1$s %3$s', 1, 2, 3)
+-- !query schema
+struct<format_string(%1$s %3$s, 1, 2, 3):string>
+-- !query output
+1 3
+
+
+-- !query
+select format_string('%1$s %12$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<format_string(%1$s %12$s, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12):string>
+-- !query output
+1 12
+
+
+-- !query
+select format_string('%1$s %4$s', 1, 2, 3)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%4$s'
+
+
+-- !query
+select format_string('%1$s %13$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%13$s'
+
+
+-- !query
+select format_string('%0$s', 'Hello')
+-- !query schema
+struct<>
+-- !query output
+java.util.IllegalFormatArgumentIndexException
+Illegal format argument index = 0
+

Review comment:
       Java 17 will throw exception
   
   ```
   java.util.IllegalFormatArgumentIndexException
   Illegal format argument index = 0
   ```
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930780743


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48258/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r721880046



##########
File path: pom.xml
##########
@@ -3431,6 +3433,26 @@
 
     </profile>
 
+    <profile>
+      <id>jdk-17</id>

Review comment:
       Can we make this active by default on Java 17? Does it cause any conflicts with other profiles currently?
   ```
         <activation>
           <jdk>[17,)</jdk>
         </activation>
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r721023222



##########
File path: core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import scala.collection.mutable
+
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config.{DRIVER_JAVA_OPTIONS, EXECUTOR_JAVA_OPTIONS, OptionalConfigEntry}
+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set(
+    "--add-opens=java.base/java.lang=ALL-UNNAMED",

Review comment:
       Right, that is clear. Take the new memory access API. We can use it, but can't only use it until Java 14 support is dropped, and that's far away. It's either support two approaches at once now, or use these flags, and the latter seems much simpler still for now. So the question is mostly how to make that least surprising for users. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719370429



##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -399,6 +399,8 @@ class SparkContext(config: SparkConf) extends Logging {
     // This should be set as early as possible.
     SparkContext.fillMissingMagicCommitterConfsIfNeeded(_conf)
 
+    JavaModuleUtils.supplementJava17ModuleOptsIfNeeded(_conf)

Review comment:
       No, this is necessary for UTs in the `core` module
   
    




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931044094


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143755/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930801330






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931215600


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143763/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930797235


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143748/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931041244


   **[Test build #143755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143755/testReport)** for PR 34153 at commit [`1761e74`](https://github.com/apache/spark/commit/1761e7474be5fdd73f9157b70f919f8ff0e0d535).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939717082


   **[Test build #144069 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144069/testReport)** for PR 34153 at commit [`f61215f`](https://github.com/apache/spark/commit/f61215f205d6702028df7d69c4e3ad0472df8999).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r727680105



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       https://github.com/apache/spark/commit/92caa751257b894887d34e6abf02307931c090cd @HyukjinKwon already reverted this @srowen , I'm trying to find out what the minimum value, maybe 4100m can work normal
   
    

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       https://github.com/apache/spark/commit/92caa751257b894887d34e6abf02307931c090cd @HyukjinKwon already reverted this, @srowen , I'm trying to find out what the minimum value, maybe 4100m can work normal
   
    




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940768052


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48603/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940169273


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144090/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940692598


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48598/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940063706


   **[Test build #144086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144086/testReport)** for PR 34153 at commit [`2925783`](https://github.com/apache/spark/commit/2925783a222e47d1ea3fa499dd711e2ce3fb46aa).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r728032743



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       ```
   test("SPARK-36464: size returns correct positive number even with over 2GB data") {
       val ref = new Array[Byte](1024 * 1024 * 1024)
       val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
       o.write(ref)
       o.write(ref)
       o.close()
       assert(o.size > 0L) // make sure it is not overflowing
       assert(o.size == ref.length.toLong * 2)
     }
   ```
   
   @srowen  can we use `ByteBuffer.allocateDirect` instead of `ByteBuffer.allocate` in the case,  I re-test it with Java 8, Java 11 and Java 17, Heap OOM will not occur again when use `ByteBuffer.allocateDirect` even if `-Xmx` is 4g.
   
   It seems that using `ByteBuffer.allocateDirect` does not change what the UT wants to test, If we accept this, I will give a pr tomorrow.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939743066


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48546/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-943303445


   **[Test build #144259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144259/testReport)** for PR 34153 at commit [`40ab4a8`](https://github.com/apache/spark/commit/40ab4a846651656777a41f0388d5f79830bf8c9b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930773052


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48256/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930774394


   **[Test build #143746 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143746/testReport)** for PR 34153 at commit [`adc566d`](https://github.com/apache/spark/commit/adc566d9db76a20899eab37ebaeb24b6df85c0d3).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941922335


   **[Test build #144172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144172/testReport)** for PR 34153 at commit [`ee8b1b8`](https://github.com/apache/spark/commit/ee8b1b8d070d168e93ae21319b2b4d9737416f74).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `        new_class = type(NameTypeHolder.short_name, (NameTypeHolder,), `
     * `class NoSuchIndexException(message: String, cause: Option[Throwable] = None)`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939967252


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144075/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930776737


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143747/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930780755


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48258/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-938019873


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48473/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-945359118


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48820/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946335498


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48857/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946319186


   **[Test build #144383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144383/testReport)** for PR 34153 at commit [`e773fc4`](https://github.com/apache/spark/commit/e773fc4e57b3fd3c2272ef4d89b3865388c10789).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r722172073



##########
File path: pom.xml
##########
@@ -3431,6 +3433,26 @@
 
     </profile>
 
+    <profile>
+      <id>jdk-17</id>

Review comment:
       I don't think that helps - this just controls whether it's active when _built_ with JDK 17. We're not building with 17 anytime soon




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940107946


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48564/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940175418


   **[Test build #144092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144092/testReport)** for PR 34153 at commit [`1fb8fb1`](https://github.com/apache/spark/commit/1fb8fb15dab8b0799a7eb3dddd3ffbd93e8ebafc).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-938019873


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48473/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939648571


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48543/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r727927446



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Hm, I'd be OK just removing the test case "SPARK-36464: size returns correct positive number even with over 2GB data". What it tests is a clean fix, and this is a huge amount of mem to allocate to test.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941928030


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144172/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939720548


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48546/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941924977


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48657/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940169273


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144090/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930775797


   **[Test build #143747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143747/testReport)** for PR 34153 at commit [`70fc3bf`](https://github.com/apache/spark/commit/70fc3bf618c909baf2eb65ab25b1aaf4376b0fea).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r728619162



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Yeah something like that. I was thinking you'd write it 1025 times to go just over the 2GB limit. But sounds like that doesn't get at the problem - it's not the array allocation? still, might be a good change! I'm not clear what's running OOM here then.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r722825382



##########
File path: core/src/test/scala/org/apache/spark/launcher/LauncherBackendSuite.scala
##########
@@ -46,15 +46,22 @@ class LauncherBackendSuite extends SparkFunSuite with Matchers {
   private def testWithMaster(master: String): Unit = {
     val env = new java.util.HashMap[String, String]()
     env.put("SPARK_PRINT_LAUNCH_COMMAND", "1")
-    val handle = new SparkLauncher(env)
+    val launcher = new SparkLauncher(env)
       .setSparkHome(sys.props("spark.test.home"))
       .setConf(SparkLauncher.DRIVER_EXTRA_CLASSPATH, System.getProperty("java.class.path"))
       .setConf(UI_ENABLED.key, "false")
       .setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS, s"-Dtest.appender=console")
       .setMaster(master)
       .setAppResource(SparkLauncher.NO_RESOURCE)
       .setMainClass(TestApp.getClass.getName().stripSuffix("$"))
-      .startApplication()
+
+    if(JavaModuleUtils.isJavaVersionAtLeast17) {

Review comment:
       Thank you, @srowen .




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-943303445


   **[Test build #144259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144259/testReport)** for PR 34153 at commit [`40ab4a8`](https://github.com/apache/spark/commit/40ab4a846651656777a41f0388d5f79830bf8c9b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719067240



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Yes, use -Xmx4g, `SPARK-36464: size returns correct positive number even with over 2GB data` will oom




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930915413


   **[Test build #143758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143758/testReport)** for PR 34153 at commit [`0861c5d`](https://github.com/apache/spark/commit/0861c5d72f1c4007a9282319f79b9b888ff7732f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931037262


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143753/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r721880795



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##########
@@ -322,6 +323,13 @@ private object DateTimeFormatterHelper {
             (isParsing && unsupportedLettersForParsing.contains(c))) {
             throw new IllegalArgumentException(s"Illegal pattern character: $c")
           }
+          // SPARK-36796: `select date_format('2018-11-17 13:33:33.333', 'B')` failed with Java 8,
+          // but use Java 17 will return `in the afternoon` because 'B' is used to represent
+          // `Pattern letters to output a day period` in Java 17 and disabled it here for
+          // compatibility with Java 8 behavior.
+          for (c <- patternPart if unknownPatternLetters.contains(c)) {

Review comment:
       Could you make this a separate PR please, @LuciferYang ? We can merge it first before this.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##########
@@ -322,6 +323,13 @@ private object DateTimeFormatterHelper {
             (isParsing && unsupportedLettersForParsing.contains(c))) {
             throw new IllegalArgumentException(s"Illegal pattern character: $c")
           }
+          // SPARK-36796: `select date_format('2018-11-17 13:33:33.333', 'B')` failed with Java 8,
+          // but use Java 17 will return `in the afternoon` because 'B' is used to represent
+          // `Pattern letters to output a day period` in Java 17 and disabled it here for
+          // compatibility with Java 8 behavior.
+          for (c <- patternPart if unknownPatternLetters.contains(c)) {

Review comment:
       Could you make this a separate PR please, @LuciferYang ? We can merge that first before this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r730543605



##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql-jdk17.out
##########
@@ -0,0 +1,360 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 42
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text string') AS true
+-- !query schema
+struct<true:boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text strin') AS `false`
+-- !query schema
+struct<false:boolean>
+-- !query output
+false
+
+
+-- !query
+CREATE TABLE TEXT_TBL (f1 string) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('doh!')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('hi de ho neighbor')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT '' AS two, * FROM TEXT_TBL
+-- !query schema
+struct<two:string,f1:string>
+-- !query output
+	doh!
+	hi de ho neighbor
+
+
+-- !query
+select length(42)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'length(42)' due to data type mismatch: argument 1 requires (string or binary) type, however, '42' is of int type.; line 1 pos 7
+
+
+-- !query
+select string('four: ') || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(CAST('four: ' AS STRING), 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 'four: ' || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat('four: ', 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 3 || 4.0
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(3, 4.0BD)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, decimal(2,1)]; line 1 pos 7
+
+
+-- !query
+/*
+ * various string functions
+ */
+select concat('one')
+-- !query schema
+struct<concat(one):string>
+-- !query output
+one
+
+
+-- !query
+select concat(1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, int, int, string, boolean, boolean, date]; line 1 pos 7
+
+
+-- !query
+select concat_ws('#','one')
+-- !query schema
+struct<concat_ws(#, one):string>
+-- !query output
+one
+
+
+-- !query
+select concat_ws('#',1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('#', 1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '1' is of int type. argument 3 requires (array<string> or string) type, however, '2' is of int type. argument 4 requires (array<string> or string) type, however, '3' is of int type. argument 6 requires (array<string> or string) type, however, 'true' is of boolean type. argument 7 requires (array<string> or string) type, however, 'false' is of boolean type. argument 8 requires (array<string> or string) type, however, 'to_date('20100309', 'yyyyMMdd')' is of date type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(',',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(',', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws('',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(NULL,10,20,null,30) is null
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(CAST(NULL AS STRING), 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select reverse('abcde')
+-- !query schema
+struct<reverse(abcde):string>
+-- !query output
+edcba
+
+
+-- !query
+select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'substring('ahoj', 1, t.i)' due to data type mismatch: argument 3 requires int type, however, 't.i' is of bigint type.; line 1 pos 10
+
+
+-- !query
+/*
+ * format
+ */
+select format_string(NULL)
+-- !query schema
+struct<format_string(NULL):string>
+-- !query output
+NULL
+
+
+-- !query
+select format_string('Hello')
+-- !query schema
+struct<format_string(Hello):string>
+-- !query output
+Hello
+
+
+-- !query
+select format_string('Hello %s', 'World')
+-- !query schema
+struct<format_string(Hello %s, World):string>
+-- !query output
+Hello World
+
+
+-- !query
+select format_string('Hello %%')
+-- !query schema
+struct<format_string(Hello %%):string>
+-- !query output
+Hello %
+
+
+-- !query
+select format_string('Hello %%%%')
+-- !query schema
+struct<format_string(Hello %%%%):string>
+-- !query output
+Hello %%
+
+
+-- !query
+select format_string('Hello %s %s', 'World')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %s')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %x', 20)
+-- !query schema
+struct<format_string(Hello %x, 20):string>
+-- !query output
+Hello 14
+
+
+-- !query
+select format_string('%1$s %3$s', 1, 2, 3)
+-- !query schema
+struct<format_string(%1$s %3$s, 1, 2, 3):string>
+-- !query output
+1 3
+
+
+-- !query
+select format_string('%1$s %12$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<format_string(%1$s %12$s, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12):string>
+-- !query output
+1 12
+
+
+-- !query
+select format_string('%1$s %4$s', 1, 2, 3)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%4$s'
+
+
+-- !query
+select format_string('%1$s %13$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%13$s'
+
+
+-- !query
+select format_string('%0$s', 'Hello')
+-- !query schema
+struct<>
+-- !query output
+java.util.IllegalFormatArgumentIndexException
+Illegal format argument index = 0
+

Review comment:
       Currently, only this case is not related with `--add-opens`, which seems to be the difference between Java 8 and Java 17




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-945377089


   **[Test build #144341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144341/testReport)** for PR 34153 at commit [`5098d31`](https://github.com/apache/spark/commit/5098d3152e3f4fa37075c58244d6df7746ad2f36).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-945317457


   **[Test build #144341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144341/testReport)** for PR 34153 at commit [`5098d31`](https://github.com/apache/spark/commit/5098d3152e3f4fa37075c58244d6df7746ad2f36).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946329059


   If this pr is merged, we can continue to verify other modules


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946319186






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-947267050


   thanks all ~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946320548


   **[Test build #144384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144384/testReport)** for PR 34153 at commit [`2dec62e`](https://github.com/apache/spark/commit/2dec62e237dae6d1d2a0b70134eac73401963320).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r731446119



##########
File path: launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.launcher;
+
+import java.util.Arrays;
+import java.util.HashSet;
+
+public class JavaModuleOptions {
+    private static final String[] DEFAULT_MODULE_OPTIONS = new String[] {
+      "-XX:+IgnoreUnrecognizedVMOptions",
+      "--add-opens=java.base/java.lang=ALL-UNNAMED",
+      "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
+      "--add-opens=java.base/java.io=ALL-UNNAMED",
+      "--add-opens=java.base/java.net=ALL-UNNAMED",
+      "--add-opens=java.base/java.nio=ALL-UNNAMED",
+      "--add-opens=java.base/java.util=ALL-UNNAMED",
+      "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
+      "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
+      "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
+      "--add-opens=java.base/sun.security.action=ALL-UNNAMED",
+      "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED"};
+
+    public static String defaultModuleOptions() {
+      return String.join(" ", new HashSet<>(Arrays.asList(DEFAULT_MODULE_OPTIONS)));

Review comment:
       OK, use `DEFAULT_MODULE_OPTIONS` directly instead
   
   

##########
File path: launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.launcher;
+
+import java.util.Arrays;
+import java.util.HashSet;
+
+public class JavaModuleOptions {
+    private static final String[] DEFAULT_MODULE_OPTIONS = new String[] {

Review comment:
       ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946354380


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48858/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930771288






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930774411


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143746/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931044094


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143755/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931110931


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48275/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931012179


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48266/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931206315


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143762/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r720637127



##########
File path: core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import scala.collection.mutable
+
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config.{DRIVER_JAVA_OPTIONS, EXECUTOR_JAVA_OPTIONS, OptionalConfigEntry}
+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set(
+    "--add-opens=java.base/java.lang=ALL-UNNAMED",

Review comment:
       Yes, @srowen . This has been warning at Java 11 ~ 16, and Java 17 enforces it finally.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931017742


   **[Test build #143751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143751/testReport)** for PR 34153 at commit [`7e016b3`](https://github.com/apache/spark/commit/7e016b32434dcdc76e1396eca95a54e41449b193).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931214251


   **[Test build #143763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143763/testReport)** for PR 34153 at commit [`4981260`](https://github.com/apache/spark/commit/498126083d2f2f42163b12a234ea53858bff0a8e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939629761


   @srowen @dongjoon-hyun Very sorry, I just finished my holiday :) 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939743066


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48546/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r725760528



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##########
@@ -322,6 +323,13 @@ private object DateTimeFormatterHelper {
             (isParsing && unsupportedLettersForParsing.contains(c))) {
             throw new IllegalArgumentException(s"Illegal pattern character: $c")
           }
+          // SPARK-36796: `select date_format('2018-11-17 13:33:33.333', 'B')` failed with Java 8,
+          // but use Java 17 will return `in the afternoon` because 'B' is used to represent
+          // `Pattern letters to output a day period` in Java 17 and disabled it here for
+          // compatibility with Java 8 behavior.
+          for (c <- patternPart if unknownPatternLetters.contains(c)) {

Review comment:
       > Could you make this a separate PR please, @LuciferYang ? We can merge that first before this.
   
   OK




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940159858






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941960535


   **[Test build #144179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144179/testReport)** for PR 34153 at commit [`bc4bad7`](https://github.com/apache/spark/commit/bc4bad72bf3c42bd420eb60a2acb26d0eae39a33).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940169219


   **[Test build #144090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144090/testReport)** for PR 34153 at commit [`0e4bba8`](https://github.com/apache/spark/commit/0e4bba8e37a1af76da8f21a49cf026abee6d5b4b).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940693876


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48598/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-938019873


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48473/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940159865






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940799455


   **[Test build #144126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144126/testReport)** for PR 34153 at commit [`2e87bde`](https://github.com/apache/spark/commit/2e87bdeff7500bbf50d18efa25b02205b48a0dff).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-943374800


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48738/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r725798422



##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql-jdk17.out
##########
@@ -0,0 +1,360 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 42
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text string') AS true
+-- !query schema
+struct<true:boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text strin') AS `false`
+-- !query schema
+struct<false:boolean>
+-- !query output
+false
+
+
+-- !query
+CREATE TABLE TEXT_TBL (f1 string) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('doh!')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('hi de ho neighbor')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT '' AS two, * FROM TEXT_TBL
+-- !query schema
+struct<two:string,f1:string>
+-- !query output
+	doh!
+	hi de ho neighbor
+
+
+-- !query
+select length(42)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'length(42)' due to data type mismatch: argument 1 requires (string or binary) type, however, '42' is of int type.; line 1 pos 7
+
+
+-- !query
+select string('four: ') || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(CAST('four: ' AS STRING), 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 'four: ' || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat('four: ', 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 3 || 4.0
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(3, 4.0BD)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, decimal(2,1)]; line 1 pos 7
+
+
+-- !query
+/*
+ * various string functions
+ */
+select concat('one')
+-- !query schema
+struct<concat(one):string>
+-- !query output
+one
+
+
+-- !query
+select concat(1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, int, int, string, boolean, boolean, date]; line 1 pos 7
+
+
+-- !query
+select concat_ws('#','one')
+-- !query schema
+struct<concat_ws(#, one):string>
+-- !query output
+one
+
+
+-- !query
+select concat_ws('#',1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('#', 1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '1' is of int type. argument 3 requires (array<string> or string) type, however, '2' is of int type. argument 4 requires (array<string> or string) type, however, '3' is of int type. argument 6 requires (array<string> or string) type, however, 'true' is of boolean type. argument 7 requires (array<string> or string) type, however, 'false' is of boolean type. argument 8 requires (array<string> or string) type, however, 'to_date('20100309', 'yyyyMMdd')' is of date type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(',',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(',', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws('',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(NULL,10,20,null,30) is null
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(CAST(NULL AS STRING), 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select reverse('abcde')
+-- !query schema
+struct<reverse(abcde):string>
+-- !query output
+edcba
+
+
+-- !query
+select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'substring('ahoj', 1, t.i)' due to data type mismatch: argument 3 requires int type, however, 't.i' is of bigint type.; line 1 pos 10
+
+
+-- !query
+/*
+ * format
+ */
+select format_string(NULL)
+-- !query schema
+struct<format_string(NULL):string>
+-- !query output
+NULL
+
+
+-- !query
+select format_string('Hello')
+-- !query schema
+struct<format_string(Hello):string>
+-- !query output
+Hello
+
+
+-- !query
+select format_string('Hello %s', 'World')
+-- !query schema
+struct<format_string(Hello %s, World):string>
+-- !query output
+Hello World
+
+
+-- !query
+select format_string('Hello %%')
+-- !query schema
+struct<format_string(Hello %%):string>
+-- !query output
+Hello %
+
+
+-- !query
+select format_string('Hello %%%%')
+-- !query schema
+struct<format_string(Hello %%%%):string>
+-- !query output
+Hello %%
+
+
+-- !query
+select format_string('Hello %s %s', 'World')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %s')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %x', 20)
+-- !query schema
+struct<format_string(Hello %x, 20):string>
+-- !query output
+Hello 14
+
+
+-- !query
+select format_string('%1$s %3$s', 1, 2, 3)
+-- !query schema
+struct<format_string(%1$s %3$s, 1, 2, 3):string>
+-- !query output
+1 3
+
+
+-- !query
+select format_string('%1$s %12$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<format_string(%1$s %12$s, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12):string>
+-- !query output
+1 12
+
+
+-- !query
+select format_string('%1$s %4$s', 1, 2, 3)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%4$s'
+
+
+-- !query
+select format_string('%1$s %13$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%13$s'
+
+
+-- !query
+select format_string('%0$s', 'Hello')
+-- !query schema
+struct<>
+-- !query output
+java.util.IllegalFormatArgumentIndexException
+Illegal format argument index = 0
+

Review comment:
       @srowen @dongjoon-hyun Does this behavior need to be consistent with Java 8?
   
   ```
   https://github.com/apache/spark/blob/128168d8c4019a1e10a9f1be734868524f6a09f0/sql/core/src/test/resources/sql-tests/inputs/postgreSQL/text.sql#L90-L91
   ```
   It seems that Java 17 is same as PostgreSQL, but Java 8 returns the a non ERROR result 
   
   ```
   https://github.com/apache/spark/blob/128168d8c4019a1e10a9f1be734868524f6a09f0/sql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql.out#L274-L279
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r728613321



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       @srowen  do you mean like the following code:
   
   ```
       val ref = new Array[Byte](1024 * 1024 * 2)
       val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
       (0 until 1024).foreach(_ => o.write(ref))
       assert(o.size > 0L) // make sure it is not overflowing
       assert(o.size == ref.length.toLong * 1024)
   ```
   use a small `ref(1g -> 2m)` and write more times(2 -> 1024)?  I tested this with Java 11 and heap OOM still occurs 
   
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r728956939



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSparkSubmitSuite.scala
##########
@@ -40,15 +40,26 @@ class WholeStageCodegenSparkSubmitSuite extends SparkSubmitTestUtils
     val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
 
     // HotSpot JVM specific: Set up a local cluster with the driver/executor using mismatched
-    // settings of UseCompressedOops JVM option.
+    // settings of UseCompressedClassPointers JVM option.

Review comment:
       If this PR resolves this too, put it in the title?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r727656443



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       @srowen already fix  SPARK-36900, this pr will no longer change the value of  `-Xmx`, I will resolve this conversation, thanks all
   
   

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Since SPARK-36900 was reverted, this problem still exists

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       https://github.com/apache/spark/commit/92caa751257b894887d34e6abf02307931c090cd @HyukjinKwon already reverted this @srowen , I'm trying to find out what the minimum value, maybe 4100m can work normal
   
    

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       https://github.com/apache/spark/commit/92caa751257b894887d34e6abf02307931c090cd @HyukjinKwon already reverted this, @srowen , I'm trying to find out what the minimum value, maybe 4100m can work normal
   
    

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       @srowen I test 
   ```
   mvn clean install -pl core -am -Dtest=none -DwildcardSuites=org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite
   ```
   with Java 11 and Java 17:
   
   - Java 11: if -Xmx >= 5155m , OOM will not occur again
   - Java 17: if -Xmx >= 4097m, OOM will not occur again
   
   
   
   

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       ```
   test("SPARK-36464: size returns correct positive number even with over 2GB data") {
       val ref = new Array[Byte](1024 * 1024 * 1024)
       val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
       o.write(ref)
       o.write(ref)
       o.close()
       assert(o.size > 0L) // make sure it is not overflowing
       assert(o.size == ref.length.toLong * 2)
     }
   ```
   
   @srowen  can we use `ByteBuffer.allocateDirect` instead of `ByteBuffer.allocate` in the case,  I test this case with Java 8, Java 11 and Java 17, Heap OOM will not occur again when use `ByteBuffer.allocateDirect` even if `-Xmx` is 4g.
   
   It seems that using `ByteBuffer.allocateDirect` does not change what the use case wants to test, If we accept this, I will give a pr tomorrow.
   
   

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       ```
   test("SPARK-36464: size returns correct positive number even with over 2GB data") {
       val ref = new Array[Byte](1024 * 1024 * 1024)
       val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
       o.write(ref)
       o.write(ref)
       o.close()
       assert(o.size > 0L) // make sure it is not overflowing
       assert(o.size == ref.length.toLong * 2)
     }
   ```
   
   @srowen  can we use `ByteBuffer.allocateDirect` instead of `ByteBuffer.allocate` in the case,  I re-test it with Java 8, Java 11 and Java 17, Heap OOM will not occur again when use `ByteBuffer.allocateDirect` even if `-Xmx` is 4g.
   
   It seems that using `ByteBuffer.allocateDirect` does not change what the use case wants to test, If we accept this, I will give a pr tomorrow.
   
   

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       ```
   test("SPARK-36464: size returns correct positive number even with over 2GB data") {
       val ref = new Array[Byte](1024 * 1024 * 1024)
       val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
       o.write(ref)
       o.write(ref)
       o.close()
       assert(o.size > 0L) // make sure it is not overflowing
       assert(o.size == ref.length.toLong * 2)
     }
   ```
   
   @srowen  can we use `ByteBuffer.allocateDirect` instead of `ByteBuffer.allocate` in the case,  I re-test it with Java 8, Java 11 and Java 17, Heap OOM will not occur again when use `ByteBuffer.allocateDirect` even if `-Xmx` is 4g.
   
   It seems that using `ByteBuffer.allocateDirect` does not change what the UT wants to test, If we accept this, I will give a pr tomorrow.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940664591


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48598/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941855485






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941955260


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48657/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946339250


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48858/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-947134802


   Thank you so much, @LuciferYang and @srowen and all!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-944927571


   I merged the other PRs - this needs a rebase now and we can test again


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931144788


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48273/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931043722


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48273/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930999186


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48269/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931120400


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48269/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-937984130






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r721880262



##########
File path: pom.xml
##########
@@ -3431,6 +3433,26 @@
 
     </profile>
 
+    <profile>
+      <id>jdk-17</id>
+      <properties>
+        <extraJavaTestArgs>
+          --add-opens java.base/java.nio=ALL-UNNAMED

Review comment:
       Please use `--add-opens=java.base/java.nio=ALL-UNNAMED` pattern.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940167010


   **[Test build #144091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144091/testReport)** for PR 34153 at commit [`ad22526`](https://github.com/apache/spark/commit/ad22526b1e21b4ce053bf7db663f23ec730aa5d8).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939646962


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144065/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939879533


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48553/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940144322


   There are failed tests. Will fix them tomorrow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940027208


   **[Test build #144084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144084/testReport)** for PR 34153 at commit [`8628ed6`](https://github.com/apache/spark/commit/8628ed65e3d982108f5b672d6e1f1fd65e59721d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940066129


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48562/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940640077


   **[Test build #144121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144121/testReport)** for PR 34153 at commit [`fdd912e`](https://github.com/apache/spark/commit/fdd912ed3f2114f57dcdca981591b87ec918a24b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931120400


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48269/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r728032743



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       ```
   test("SPARK-36464: size returns correct positive number even with over 2GB data") {
       val ref = new Array[Byte](1024 * 1024 * 1024)
       val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
       o.write(ref)
       o.write(ref)
       o.close()
       assert(o.size > 0L) // make sure it is not overflowing
       assert(o.size == ref.length.toLong * 2)
     }
   ```
   
   @srowen  can we use `ByteBuffer.allocateDirect` instead of `ByteBuffer.allocate` in the case,  I test this case with Java 8, Java 11 and Java 17, Heap OOM will not occur again when use `ByteBuffer.allocateDirect` even if `-Xmx` is 4g.
   
   It seems that using `ByteBuffer.allocateDirect` does not change what the use case wants to test, If we accept this, I will give a pr tomorrow.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r727656443



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       @srowen already fix  SPARK-36900, this pr will no longer change the value of  `-Xmx`, I will resolve this conversation, thanks all
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930916198


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48262/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r720688637



##########
File path: core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import scala.collection.mutable
+
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config.{DRIVER_JAVA_OPTIONS, EXECUTOR_JAVA_OPTIONS, OptionalConfigEntry}
+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set(
+    "--add-opens=java.base/java.lang=ALL-UNNAMED",

Review comment:
       @dongjoon-hyun @srowen Or can we configure a comprehensive set of `--add-opens` args for users by default (not only for UTs), and add `-XX:+IgnoreUnrecognizedVMOptions`  at the same time? maybe this can reduce the use cost with JDK 17 of users. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930953448


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48265/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931191523






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930780755


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48258/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946360012






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946357248


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48857/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946375350


   **[Test build #144383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144383/testReport)** for PR 34153 at commit [`e773fc4`](https://github.com/apache/spark/commit/e773fc4e57b3fd3c2272ef4d89b3865388c10789).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946415929


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48871/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946454739


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48871/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946381465






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946381465






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930770855






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719111732



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Create SPARK-36900




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930776737


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143747/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719053735



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Does this mean JDK17 require more memory? Otherwise, please revert this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930987724


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48266/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940700519


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144121/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940694569


   **[Test build #144126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144126/testReport)** for PR 34153 at commit [`2e87bde`](https://github.com/apache/spark/commit/2e87bdeff7500bbf50d18efa25b02205b48a0dff).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940196155


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48569/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r725795555



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSparkSubmitSuite.scala
##########
@@ -40,15 +40,26 @@ class WholeStageCodegenSparkSubmitSuite extends SparkSubmitTestUtils
     val unusedJar = TestUtils.createJarWithClasses(Seq.empty)
 
     // HotSpot JVM specific: Set up a local cluster with the driver/executor using mismatched
-    // settings of UseCompressedOops JVM option.
+    // settings of UseCompressedClassPointers JVM option.
     val argsForSparkSubmit = Seq(
       "--class", WholeStageCodegenSparkSubmitSuite.getClass.getName.stripSuffix("$"),
       "--master", "local-cluster[1,1,1024]",
       "--driver-memory", "1g",
       "--conf", "spark.ui.enabled=false",
       "--conf", "spark.master.rest.enabled=false",
-      "--conf", "spark.driver.extraJavaOptions=-XX:-UseCompressedOops",
-      "--conf", "spark.executor.extraJavaOptions=-XX:+UseCompressedOops",
+      // SPARK-36796: The results of `Platform.BYTE_ARRAY_OFFSET` using different Java versions

Review comment:
       @dongjoon-hyun @srowen What do you think about this case? In Java 8, `UseCompressedOops` and `UseCompressedClassPointers` are both passed. But In Java 17, `UseCompressedClassPointers` should be used.  Should make this a separate PR? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939646962


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144065/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r727677263



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       It was left at 5g - did that not work for the test?

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Hm. So we need more memory, or else figure out why more memory is needed. This change might have caused other flakiness; if that's not the case we'll just increase memory again. Otherwise I think we have to 'fix' this test.
   
   In Java 9+ we changed some of the code that handles DirectByteBuffer to not do some things with reflection if the classes were not opened up with command line flags. I wonder if that's a difference, but, I think these flags now open up those classes, so would have expected this works like Java 8 again (unless more has to be opened up). 
   
   But maybe just needs a test change.

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Hm, I'd be OK just removing the test case "SPARK-36464: size returns correct positive number even with over 2GB data". What it tests is a clean fix, and this is a huge amount of mem to allocate to test.

##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       That could be fine. Actually could we just allocate an array 1000 times smaller and write it 1001 times? that kind of thing. That seems like it would also achieve the same effect.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r728032743



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       ```
   test("SPARK-36464: size returns correct positive number even with over 2GB data") {
       val ref = new Array[Byte](1024 * 1024 * 1024)
       val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
       o.write(ref)
       o.write(ref)
       o.close()
       assert(o.size > 0L) // make sure it is not overflowing
       assert(o.size == ref.length.toLong * 2)
     }
   ```
   
   @srowen  can we use `ByteBuffer.allocateDirect` instead of `ByteBuffer.allocate` in the case,  I re-test it with Java 8, Java 11 and Java 17, Heap OOM will not occur again when use `ByteBuffer.allocateDirect` even if `-Xmx` is 4g.
   
   It seems that using `ByteBuffer.allocateDirect` does not change what the use case wants to test, If we accept this, I will give a pr tomorrow.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941961619


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144179/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-943448340


   **[Test build #144259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144259/testReport)** for PR 34153 at commit [`40ab4a8`](https://github.com/apache/spark/commit/40ab4a846651656777a41f0388d5f79830bf8c9b).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `class HistogramPlotBase(NumericPlotBase):`
     * `class KdePlotBase(NumericPlotBase):`
     * `class SparkUpgradeException(CapturedException):`
     * `public class ExpressionImplUtils `
     * `case class AesEncrypt(input: Expression, key: Expression, child: Expression)`
     * `case class AesDecrypt(input: Expression, key: Expression, child: Expression)`
     * `case class CreateView(`
     * `case class ShowCatalogsCommand(pattern: Option[String]) extends LeafRunnableCommand `


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940192000


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48568/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931015672


   **[Test build #143763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143763/testReport)** for PR 34153 at commit [`4981260`](https://github.com/apache/spark/commit/498126083d2f2f42163b12a234ea53858bff0a8e).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931175314


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48275/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939846149


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144068/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940063706


   **[Test build #144086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144086/testReport)** for PR 34153 at commit [`2925783`](https://github.com/apache/spark/commit/2925783a222e47d1ea3fa499dd711e2ce3fb46aa).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941940728


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48657/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r720620194



##########
File path: core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import scala.collection.mutable
+
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config.{DRIVER_JAVA_OPTIONS, EXECUTOR_JAVA_OPTIONS, OptionalConfigEntry}
+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set(
+    "--add-opens=java.base/java.lang=ALL-UNNAMED",

Review comment:
       Yes，Spark doesn't work without these. It seems that JDK 17 enforce this restriction now. I haven't found a workaround way to prevent these exceptions from being thrown. Let me investigate again.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931034047


   **[Test build #143753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143753/testReport)** for PR 34153 at commit [`44c566f`](https://github.com/apache/spark/commit/44c566f6541ddb00e3a1b6238e803256f7f83f7d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r730543605



##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql-jdk17.out
##########
@@ -0,0 +1,360 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 42
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text string') AS true
+-- !query schema
+struct<true:boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text strin') AS `false`
+-- !query schema
+struct<false:boolean>
+-- !query output
+false
+
+
+-- !query
+CREATE TABLE TEXT_TBL (f1 string) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('doh!')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('hi de ho neighbor')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT '' AS two, * FROM TEXT_TBL
+-- !query schema
+struct<two:string,f1:string>
+-- !query output
+	doh!
+	hi de ho neighbor
+
+
+-- !query
+select length(42)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'length(42)' due to data type mismatch: argument 1 requires (string or binary) type, however, '42' is of int type.; line 1 pos 7
+
+
+-- !query
+select string('four: ') || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(CAST('four: ' AS STRING), 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 'four: ' || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat('four: ', 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 3 || 4.0
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(3, 4.0BD)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, decimal(2,1)]; line 1 pos 7
+
+
+-- !query
+/*
+ * various string functions
+ */
+select concat('one')
+-- !query schema
+struct<concat(one):string>
+-- !query output
+one
+
+
+-- !query
+select concat(1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, int, int, string, boolean, boolean, date]; line 1 pos 7
+
+
+-- !query
+select concat_ws('#','one')
+-- !query schema
+struct<concat_ws(#, one):string>
+-- !query output
+one
+
+
+-- !query
+select concat_ws('#',1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('#', 1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '1' is of int type. argument 3 requires (array<string> or string) type, however, '2' is of int type. argument 4 requires (array<string> or string) type, however, '3' is of int type. argument 6 requires (array<string> or string) type, however, 'true' is of boolean type. argument 7 requires (array<string> or string) type, however, 'false' is of boolean type. argument 8 requires (array<string> or string) type, however, 'to_date('20100309', 'yyyyMMdd')' is of date type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(',',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(',', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws('',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(NULL,10,20,null,30) is null
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(CAST(NULL AS STRING), 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select reverse('abcde')
+-- !query schema
+struct<reverse(abcde):string>
+-- !query output
+edcba
+
+
+-- !query
+select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'substring('ahoj', 1, t.i)' due to data type mismatch: argument 3 requires int type, however, 't.i' is of bigint type.; line 1 pos 10
+
+
+-- !query
+/*
+ * format
+ */
+select format_string(NULL)
+-- !query schema
+struct<format_string(NULL):string>
+-- !query output
+NULL
+
+
+-- !query
+select format_string('Hello')
+-- !query schema
+struct<format_string(Hello):string>
+-- !query output
+Hello
+
+
+-- !query
+select format_string('Hello %s', 'World')
+-- !query schema
+struct<format_string(Hello %s, World):string>
+-- !query output
+Hello World
+
+
+-- !query
+select format_string('Hello %%')
+-- !query schema
+struct<format_string(Hello %%):string>
+-- !query output
+Hello %
+
+
+-- !query
+select format_string('Hello %%%%')
+-- !query schema
+struct<format_string(Hello %%%%):string>
+-- !query output
+Hello %%
+
+
+-- !query
+select format_string('Hello %s %s', 'World')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %s')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %x', 20)
+-- !query schema
+struct<format_string(Hello %x, 20):string>
+-- !query output
+Hello 14
+
+
+-- !query
+select format_string('%1$s %3$s', 1, 2, 3)
+-- !query schema
+struct<format_string(%1$s %3$s, 1, 2, 3):string>
+-- !query output
+1 3
+
+
+-- !query
+select format_string('%1$s %12$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<format_string(%1$s %12$s, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12):string>
+-- !query output
+1 12
+
+
+-- !query
+select format_string('%1$s %4$s', 1, 2, 3)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%4$s'
+
+
+-- !query
+select format_string('%1$s %13$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%13$s'
+
+
+-- !query
+select format_string('%0$s', 'Hello')
+-- !query schema
+struct<>
+-- !query output
+java.util.IllegalFormatArgumentIndexException
+Illegal format argument index = 0
+

Review comment:
       Currently, only this case is not associated with `--add-opens`, which seems to be the difference between Java 8 and Java 17




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-945359138


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48820/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-945359138


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48820/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-945378904


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144341/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931144728


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48273/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931022817


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143751/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939714729


   **[Test build #144068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144068/testReport)** for PR 34153 at commit [`733a754`](https://github.com/apache/spark/commit/733a754b00f91548d8ea4e115d02dff9e71fa15d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939834081


   **[Test build #144075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144075/testReport)** for PR 34153 at commit [`e70e4cc`](https://github.com/apache/spark/commit/e70e4cc5a08ec61f1b027bc4ad3435dabdc586de).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940131641


   **[Test build #144084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144084/testReport)** for PR 34153 at commit [`8628ed6`](https://github.com/apache/spark/commit/8628ed65e3d982108f5b672d6e1f1fd65e59721d).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `public class JavaModuleOptions `


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939659783


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48543/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940112361


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48562/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930796238


   **[Test build #143748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143748/testReport)** for PR 34153 at commit [`2b60264`](https://github.com/apache/spark/commit/2b6026460e56d53810b0a8c3df7f01cf4044a996).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930773439


   **[Test build #143746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143746/testReport)** for PR 34153 at commit [`adc566d`](https://github.com/apache/spark/commit/adc566d9db76a20899eab37ebaeb24b6df85c0d3).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r720859389



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##########
@@ -322,6 +323,13 @@ private object DateTimeFormatterHelper {
             (isParsing && unsupportedLettersForParsing.contains(c))) {
             throw new IllegalArgumentException(s"Illegal pattern character: $c")
           }
+          // SPARK-36796: `select date_format('2018-11-17 13:33:33.333', 'B')` failed with Java 8,
+          // but use Java 17 will return `in the afternoon` because 'B' is used to represent
+          // `Pattern letters to output a day period` in Java 17 and disabled it here for
+          // compatibility with Java 8 behavior.
+          for (c <- patternPart if unknownPatternLetters.contains(c)) {

Review comment:
       @MaxGekk 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719357131



##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -399,6 +399,8 @@ class SparkContext(config: SparkConf) extends Logging {
     // This should be set as early as possible.
     SparkContext.fillMissingMagicCommitterConfsIfNeeded(_conf)
 
+    JavaModuleUtils.supplementJava17ModuleOptsIfNeeded(_conf)

Review comment:
       This call may not be necessary. I'm testing it
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r720859545



##########
File path: core/src/test/scala/org/apache/spark/launcher/LauncherBackendSuite.scala
##########
@@ -46,15 +46,22 @@ class LauncherBackendSuite extends SparkFunSuite with Matchers {
   private def testWithMaster(master: String): Unit = {
     val env = new java.util.HashMap[String, String]()
     env.put("SPARK_PRINT_LAUNCH_COMMAND", "1")
-    val handle = new SparkLauncher(env)
+    val launcher = new SparkLauncher(env)
       .setSparkHome(sys.props("spark.test.home"))
       .setConf(SparkLauncher.DRIVER_EXTRA_CLASSPATH, System.getProperty("java.class.path"))
       .setConf(UI_ENABLED.key, "false")
       .setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS, s"-Dtest.appender=console")
       .setMaster(master)
       .setAppResource(SparkLauncher.NO_RESOURCE)
       .setMainClass(TestApp.getClass.getName().stripSuffix("$"))
-      .startApplication()
+
+    if(JavaModuleUtils.isJavaVersionAtLeast17) {

Review comment:
       Rather than just add these in tests, can we ensure they get added for Java 17 in the non-test code paths? like ensure that these args are always added to the JVM command? that would go a long way to not disrupting users




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719054805



##########
File path: core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import scala.collection.mutable
+
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config.{DRIVER_JAVA_OPTIONS, EXECUTOR_JAVA_OPTIONS, OptionalConfigEntry}
+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set("--add-opens java.base/java.nio=ALL-UNNAMED",

Review comment:
       Let's use `=` instead of `space` like `--add-opens=java.base/java.nio=ALL-UNNAMED`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r722172408



##########
File path: core/src/test/scala/org/apache/spark/launcher/LauncherBackendSuite.scala
##########
@@ -46,15 +46,22 @@ class LauncherBackendSuite extends SparkFunSuite with Matchers {
   private def testWithMaster(master: String): Unit = {
     val env = new java.util.HashMap[String, String]()
     env.put("SPARK_PRINT_LAUNCH_COMMAND", "1")
-    val handle = new SparkLauncher(env)
+    val launcher = new SparkLauncher(env)
       .setSparkHome(sys.props("spark.test.home"))
       .setConf(SparkLauncher.DRIVER_EXTRA_CLASSPATH, System.getProperty("java.class.path"))
       .setConf(UI_ENABLED.key, "false")
       .setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS, s"-Dtest.appender=console")
       .setMaster(master)
       .setAppResource(SparkLauncher.NO_RESOURCE)
       .setMainClass(TestApp.getClass.getName().stripSuffix("$"))
-      .startApplication()
+
+    if(JavaModuleUtils.isJavaVersionAtLeast17) {

Review comment:
       And nit: space after if




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r721312517



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##########
@@ -322,6 +323,13 @@ private object DateTimeFormatterHelper {
             (isParsing && unsupportedLettersForParsing.contains(c))) {
             throw new IllegalArgumentException(s"Illegal pattern character: $c")
           }
+          // SPARK-36796: `select date_format('2018-11-17 13:33:33.333', 'B')` failed with Java 8,
+          // but use Java 17 will return `in the afternoon` because 'B' is used to represent
+          // `Pattern letters to output a day period` in Java 17 and disabled it here for
+          // compatibility with Java 8 behavior.
+          for (c <- patternPart if unknownPatternLetters.contains(c)) {

Review comment:
       SGTM since the pattern is not documented/supported http://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719128295



##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql-jdk17.out
##########
@@ -0,0 +1,360 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 42
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text string') AS true
+-- !query schema
+struct<true:boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text strin') AS `false`
+-- !query schema
+struct<false:boolean>
+-- !query output
+false
+
+
+-- !query
+CREATE TABLE TEXT_TBL (f1 string) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('doh!')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('hi de ho neighbor')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT '' AS two, * FROM TEXT_TBL
+-- !query schema
+struct<two:string,f1:string>
+-- !query output
+	doh!
+	hi de ho neighbor
+
+
+-- !query
+select length(42)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'length(42)' due to data type mismatch: argument 1 requires (string or binary) type, however, '42' is of int type.; line 1 pos 7
+
+
+-- !query
+select string('four: ') || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(CAST('four: ' AS STRING), 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 'four: ' || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat('four: ', 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 3 || 4.0
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(3, 4.0BD)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, decimal(2,1)]; line 1 pos 7
+
+
+-- !query
+/*
+ * various string functions
+ */
+select concat('one')
+-- !query schema
+struct<concat(one):string>
+-- !query output
+one
+
+
+-- !query
+select concat(1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, int, int, string, boolean, boolean, date]; line 1 pos 7
+
+
+-- !query
+select concat_ws('#','one')
+-- !query schema
+struct<concat_ws(#, one):string>
+-- !query output
+one
+
+
+-- !query
+select concat_ws('#',1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('#', 1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '1' is of int type. argument 3 requires (array<string> or string) type, however, '2' is of int type. argument 4 requires (array<string> or string) type, however, '3' is of int type. argument 6 requires (array<string> or string) type, however, 'true' is of boolean type. argument 7 requires (array<string> or string) type, however, 'false' is of boolean type. argument 8 requires (array<string> or string) type, however, 'to_date('20100309', 'yyyyMMdd')' is of date type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(',',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(',', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws('',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(NULL,10,20,null,30) is null
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(CAST(NULL AS STRING), 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select reverse('abcde')
+-- !query schema
+struct<reverse(abcde):string>
+-- !query output
+edcba
+
+
+-- !query
+select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'substring('ahoj', 1, t.i)' due to data type mismatch: argument 3 requires int type, however, 't.i' is of bigint type.; line 1 pos 10
+
+
+-- !query
+/*
+ * format
+ */
+select format_string(NULL)
+-- !query schema
+struct<format_string(NULL):string>
+-- !query output
+NULL
+
+
+-- !query
+select format_string('Hello')
+-- !query schema
+struct<format_string(Hello):string>
+-- !query output
+Hello
+
+
+-- !query
+select format_string('Hello %s', 'World')
+-- !query schema
+struct<format_string(Hello %s, World):string>
+-- !query output
+Hello World
+
+
+-- !query
+select format_string('Hello %%')
+-- !query schema
+struct<format_string(Hello %%):string>
+-- !query output
+Hello %
+
+
+-- !query
+select format_string('Hello %%%%')
+-- !query schema
+struct<format_string(Hello %%%%):string>
+-- !query output
+Hello %%
+
+
+-- !query
+select format_string('Hello %s %s', 'World')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %s')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %x', 20)
+-- !query schema
+struct<format_string(Hello %x, 20):string>
+-- !query output
+Hello 14
+
+
+-- !query
+select format_string('%1$s %3$s', 1, 2, 3)
+-- !query schema
+struct<format_string(%1$s %3$s, 1, 2, 3):string>
+-- !query output
+1 3
+
+
+-- !query
+select format_string('%1$s %12$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<format_string(%1$s %12$s, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12):string>
+-- !query output
+1 12
+
+
+-- !query
+select format_string('%1$s %4$s', 1, 2, 3)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%4$s'
+
+
+-- !query
+select format_string('%1$s %13$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%13$s'
+
+
+-- !query
+select format_string('%0$s', 'Hello')
+-- !query schema
+struct<>
+-- !query output
+java.util.IllegalFormatArgumentIndexException
+Illegal format argument index = 0
+

Review comment:
       @wangyum @dongjoon-hyun 
   ```
   --PostgreSQL throw ERROR:  format specifies argument 0, but arguments are numbered from 1
   select format_string('%0$s', 'Hello');
   ```
   The SQL in `postgreSQL/text.sql` has different behavior between Java8 and Java17, but it seems that the behavior of Java 17 is expected, so do Independent verification here.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719124756



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##########
@@ -322,6 +323,13 @@ private object DateTimeFormatterHelper {
             (isParsing && unsupportedLettersForParsing.contains(c))) {
             throw new IllegalArgumentException(s"Illegal pattern character: $c")
           }
+          // SPARK-36796: `select date_format('2018-11-17 13:33:33.333', 'B')` failed with Java 8,
+          // but use Java 17 will return `in the afternoon` because 'B' is used to represent
+          // `Pattern letters to output a day period` in Java 17 and disabled it here for
+          // compatibility with Java 8 behavior.
+          for (c <- patternPart if unknownPatternLetters.contains(c)) {

Review comment:
       @dongjoon-hyun  @wangyum the result of  `select date_format('2018-11-17 13:33:33.333', 'B')` in `datetime-formatting-invalid.sql` with Java 8 is:
   ```
   -- !query
   select date_format('2018-11-17 13:33:33.333', 'B')
   -- !query schema
   struct<>
   -- !query output
   java.lang.IllegalArgumentException
   Unknown pattern letter: B
   ```
   
   With Java 17 the result is  
   ```
   - datetime-formatting-invalid.sql *** FAILED ***
     datetime-formatting-invalid.sql
     Expected "struct<[]>", but got "struct<[date_format(2018-11-17 13:33:33.333, B):string]>" Schema did not match for query #34
     select date_format('2018-11-17 13:33:33.333', 'B'): -- !query
     select date_format('2018-11-17 13:33:33.333', 'B')
     -- !query schema
     struct<date_format(2018-11-17 13:33:33.333, B):string>
     -- !query output
     in the afternoon (SQLQueryTestSuite.scala:469)
   ```
   
   'B' is used to represent `Pattern letters to output a day period` in Java 17 
   
   ```
   *  Pattern  Count  Equivalent builder methods
        *  -------  -----  --------------------------
        *    B       1      appendDayPeriodText(TextStyle.SHORT)
        *    BBBB    4      appendDayPeriodText(TextStyle.FULL)
        *    BBBBB   5      appendDayPeriodText(TextStyle.NARROW)
   ```
   
   Manual disabled it there for compatibility with Java 8 behavior.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939714729


   **[Test build #144068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144068/testReport)** for PR 34153 at commit [`733a754`](https://github.com/apache/spark/commit/733a754b00f91548d8ea4e115d02dff9e71fa15d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939643060


   **[Test build #144065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144065/testReport)** for PR 34153 at commit [`4aa3643`](https://github.com/apache/spark/commit/4aa36437d07c152546975c6e32278fe2b605759f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930844882


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48262/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931111915


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48269/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719108528



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Our Jenkins job has failed for a long time:
   https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7-jdk-11/
   
   ```
   *** RUN ABORTED ***
     java.lang.OutOfMemoryError: Java heap space
     at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
     at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
     at org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite.$anonfun$new$23(ChunkedByteBufferOutputStreamSuite.scala:125)
     at org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite.$anonfun$new$23$adapted(ChunkedByteBufferOutputStreamSuite.scala:125)
     at org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite$$Lambda$9510/0x00000008427c4840.apply(Unknown Source)
     at org.apache.spark.util.io.ChunkedByteBufferOutputStream.allocateNewChunkIfNeeded(ChunkedByteBufferOutputStream.scala:87)
     at org.apache.spark.util.io.ChunkedByteBufferOutputStream.write(ChunkedByteBufferOutputStream.scala:75)
     at java.base/java.io.OutputStream.write(OutputStream.java:122)
     at org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite.$anonfun$new$22(ChunkedByteBufferOutputStreamSuite.scala:127)
     at org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite$$Lambda$1002/0x0000000840913840.apply(Unknown Source)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-943458796


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144259/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941901966






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930828991


   **[Test build #143755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143755/testReport)** for PR 34153 at commit [`1761e74`](https://github.com/apache/spark/commit/1761e7474be5fdd73f9157b70f919f8ff0e0d535).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930801330


   **[Test build #143751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143751/testReport)** for PR 34153 at commit [`7e016b3`](https://github.com/apache/spark/commit/7e016b32434dcdc76e1396eca95a54e41449b193).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930798615


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48259/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-943458796


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144259/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941865380


   ee8b1b8 merge  SPARK-36970 into this pr
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r727907738



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       @srowen I test 
   ```
   mvn clean install -pl core -am -Dtest=none -DwildcardSuites=org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite
   ```
   with Java 11 and Java 17:
   
   - Java 11: if -Xmx >= 5155m , OOM will not occur again
   - Java 17: if -Xmx >= 4097m, OOM will not occur again
   
   
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939717082


   **[Test build #144069 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144069/testReport)** for PR 34153 at commit [`f61215f`](https://github.com/apache/spark/commit/f61215f205d6702028df7d69c4e3ad0472df8999).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940817170


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144126/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941955260


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48657/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946327597


   > Do we need to wait on a resolution to the date difference here or treat it separately? would it mean we don't have to change the expected output?
   
   I created a new Jira  SPARK-37013 and https://github.com/apache/spark/pull/34153 try to fix the different string format behavior between Java 8 and Java 17, the relevant code has been removed from this pr.
   
   At present, this pr only adds the default `--add-opens` options and `-XX:+IgnoreUnrecognizedVMOptions` where needed, and passed UTs except `select format_string('%0$s', 'Hello')` in `postgreSQL/text.sql`
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r730542810



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       https://github.com/apache/spark/pull/34284 fix this




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946360012






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-946454763


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48871/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen closed pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 except one case in `postgreSQL/text.sql`

Posted by GitBox <gi...@apache.org>.
srowen closed pull request #34153:
URL: https://github.com/apache/spark/pull/34153


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931012179






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930796238


   **[Test build #143748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143748/testReport)** for PR 34153 at commit [`2b60264`](https://github.com/apache/spark/commit/2b6026460e56d53810b0a8c3df7f01cf4044a996).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930797235






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719069688



##########
File path: core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import scala.collection.mutable
+
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config.{DRIVER_JAVA_OPTIONS, EXECUTOR_JAVA_OPTIONS, OptionalConfigEntry}
+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set("--add-opens java.base/java.nio=ALL-UNNAMED",

Review comment:
       ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930775606


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48257/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930773077


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48256/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719054599



##########
File path: core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import scala.collection.mutable
+
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config.{DRIVER_JAVA_OPTIONS, EXECUTOR_JAVA_OPTIONS, OptionalConfigEntry}
+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set("--add-opens java.base/java.nio=ALL-UNNAMED",
+    "--add-opens java.base/sun.nio.ch=ALL-UNNAMED",
+    "--add-opens java.base/java.lang.invoke=ALL-UNNAMED",
+    "--add-opens java.base/java.nio=ALL-UNNAMED",
+    "--add-opens java.base/sun.nio.ch=ALL-UNNAMED",
+    "--add-opens java.base/java.lang.invoke=ALL-UNNAMED",
+    "--add-opens java.base/java.util=ALL-UNNAMED",
+    "--add-opens java.base/sun.security.action=ALL-UNNAMED",
+    "--add-opens java.base/sun.util.calendar=ALL-UNNAMED",
+    "--add-opens java.base/java.lang=ALL-UNNAMED",
+    "--add-opens java.base/sun.nio.cs=ALL-UNNAMED",
+    "--add-opens java.base/java.net=ALL-UNNAMED",
+    "--add-opens java.base/java.io=ALL-UNNAMED",
+    "--add-opens java.base/java.util.concurrent=ALL-UNNAMED",
+    "--add-exports java.base/jdk.internal.util.random=ALL-UNNAMED")

Review comment:
       BTW, I didn't realized that we need `--add-exports`. Could you explain a little more about this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930771276


   **[Test build #143745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143745/testReport)** for PR 34153 at commit [`09fcb3e`](https://github.com/apache/spark/commit/09fcb3eda1797d1d8c039255c7c5032cf1be9ab2).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r721878418



##########
File path: core/src/test/scala/org/apache/spark/launcher/LauncherBackendSuite.scala
##########
@@ -46,15 +46,22 @@ class LauncherBackendSuite extends SparkFunSuite with Matchers {
   private def testWithMaster(master: String): Unit = {
     val env = new java.util.HashMap[String, String]()
     env.put("SPARK_PRINT_LAUNCH_COMMAND", "1")
-    val handle = new SparkLauncher(env)
+    val launcher = new SparkLauncher(env)
       .setSparkHome(sys.props("spark.test.home"))
       .setConf(SparkLauncher.DRIVER_EXTRA_CLASSPATH, System.getProperty("java.class.path"))
       .setConf(UI_ENABLED.key, "false")
       .setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS, s"-Dtest.appender=console")
       .setMaster(master)
       .setAppResource(SparkLauncher.NO_RESOURCE)
       .setMainClass(TestApp.getClass.getName().stripSuffix("$"))
-      .startApplication()
+
+    if(JavaModuleUtils.isJavaVersionAtLeast17) {

Review comment:
       I agree with @LuciferYang (https://github.com/apache/spark/pull/34153#discussion_r720688637 ).
   @srowen , why don't we add these options `-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.nio=ALL-UNNAMED ...` always for all JDKs at Apache Spark 3.3? We can remove them later.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931187317


   **[Test build #143758 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143758/testReport)** for PR 34153 at commit [`0861c5d`](https://github.com/apache/spark/commit/0861c5d72f1c4007a9282319f79b9b888ff7732f).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719346725



##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -399,6 +399,8 @@ class SparkContext(config: SparkConf) extends Logging {
     // This should be set as early as possible.
     SparkContext.fillMissingMagicCommitterConfsIfNeeded(_conf)
 
+    JavaModuleUtils.supplementJava17ModuleOptsIfNeeded(_conf)

Review comment:
       I'm investigating whether can remove this call




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r720688637



##########
File path: core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import scala.collection.mutable
+
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config.{DRIVER_JAVA_OPTIONS, EXECUTOR_JAVA_OPTIONS, OptionalConfigEntry}
+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set(
+    "--add-opens=java.base/java.lang=ALL-UNNAMED",

Review comment:
       @dongjoon-hyun @srowen Or we can configure a comprehensive set of `--add-opens` args for users by default (not only for UTs), and add `-XX:+IgnoreUnrecognizedVMOptions`  at the same time,  maybe this can reduce the use cost with JDK 17 of users. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930941275


   **[Test build #143762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143762/testReport)** for PR 34153 at commit [`86517e6`](https://github.com/apache/spark/commit/86517e6f6bf4d88ed30cab78635c7c46d2f119c3).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931015672


   **[Test build #143763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143763/testReport)** for PR 34153 at commit [`4981260`](https://github.com/apache/spark/commit/498126083d2f2f42163b12a234ea53858bff0a8e).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931206315


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143762/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940171107


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144086/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940210047


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48570/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939850657


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144069/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940112361


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48562/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940027208


   **[Test build #144084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144084/testReport)** for PR 34153 at commit [`8628ed6`](https://github.com/apache/spark/commit/8628ed65e3d982108f5b672d6e1f1fd65e59721d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940746731


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48603/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940768052


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48603/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940699665


   **[Test build #144121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144121/testReport)** for PR 34153 at commit [`fdd912e`](https://github.com/apache/spark/commit/fdd912ed3f2114f57dcdca981591b87ec918a24b).
    * This patch **fails Spark unit tests**.
    * This patch **does not merge cleanly**.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719069451



##########
File path: core/src/main/scala/org/apache/spark/util/JavaModuleUtils.scala
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import scala.collection.mutable
+
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.config.{DRIVER_JAVA_OPTIONS, EXECUTOR_JAVA_OPTIONS, OptionalConfigEntry}
+
+object JavaModuleUtils {
+
+  private val javaModuleOptions = Set("--add-opens java.base/java.nio=ALL-UNNAMED",
+    "--add-opens java.base/sun.nio.ch=ALL-UNNAMED",
+    "--add-opens java.base/java.lang.invoke=ALL-UNNAMED",
+    "--add-opens java.base/java.nio=ALL-UNNAMED",
+    "--add-opens java.base/sun.nio.ch=ALL-UNNAMED",
+    "--add-opens java.base/java.lang.invoke=ALL-UNNAMED",
+    "--add-opens java.base/java.util=ALL-UNNAMED",
+    "--add-opens java.base/sun.security.action=ALL-UNNAMED",
+    "--add-opens java.base/sun.util.calendar=ALL-UNNAMED",
+    "--add-opens java.base/java.lang=ALL-UNNAMED",
+    "--add-opens java.base/sun.nio.cs=ALL-UNNAMED",
+    "--add-opens java.base/java.net=ALL-UNNAMED",
+    "--add-opens java.base/java.io=ALL-UNNAMED",
+    "--add-opens java.base/java.util.concurrent=ALL-UNNAMED",
+    "--add-exports java.base/jdk.internal.util.random=ALL-UNNAMED")

Review comment:
       `--add-exports` is useless for the current pr ,  already remove it.
   
   But It is needed in modules `mllib-local` and `mllib`, some code in test like `mock[java.util.Random]` need this `--add-exports`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939967252


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144075/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-940100482


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48562/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939923475


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48553/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941855485


   **[Test build #144172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144172/testReport)** for PR 34153 at commit [`ee8b1b8`](https://github.com/apache/spark/commit/ee8b1b8d070d168e93ae21319b2b4d9737416f74).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941901966


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48650/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-945317457


   **[Test build #144341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144341/testReport)** for PR 34153 at commit [`5098d31`](https://github.com/apache/spark/commit/5098d3152e3f4fa37075c58244d6df7746ad2f36).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-945314644


   @srowen ok ~
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-943402472


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48738/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-943402472


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48738/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-941855485






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r727673169



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       Since SPARK-36900 was reverted, this problem still exists




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r728613321



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       @srowen  do you mean like the following code:
   
   ```
       val ref = new Array[Byte](1024 * 1024 * 2)
       val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
       (0 until 1024).foreach(_ => o.write(ref))
       assert(o.size > 0L) // make sure it is not overflowing
       assert(o.size == ref.length.toLong * 1024)
   ```
   use a small `ref(1g -> 2m)` and write more times(2 -> 1024)?  I tested this with Java 11 and heap OOM still occurs in
   
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930804172


   **[Test build #143753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143753/testReport)** for PR 34153 at commit [`44c566f`](https://github.com/apache/spark/commit/44c566f6541ddb00e3a1b6238e803256f7f83f7d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r725798422



##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql-jdk17.out
##########
@@ -0,0 +1,360 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 42
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text string') AS true
+-- !query schema
+struct<true:boolean>
+-- !query output
+true
+
+
+-- !query
+SELECT string('this is a text string') = string('this is a text strin') AS `false`
+-- !query schema
+struct<false:boolean>
+-- !query output
+false
+
+
+-- !query
+CREATE TABLE TEXT_TBL (f1 string) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('doh!')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO TEXT_TBL VALUES ('hi de ho neighbor')
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT '' AS two, * FROM TEXT_TBL
+-- !query schema
+struct<two:string,f1:string>
+-- !query output
+	doh!
+	hi de ho neighbor
+
+
+-- !query
+select length(42)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'length(42)' due to data type mismatch: argument 1 requires (string or binary) type, however, '42' is of int type.; line 1 pos 7
+
+
+-- !query
+select string('four: ') || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(CAST('four: ' AS STRING), 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 'four: ' || 2+2
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat('four: ', 2)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [string, int]; line 1 pos 7
+
+
+-- !query
+select 3 || 4.0
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(3, 4.0BD)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, decimal(2,1)]; line 1 pos 7
+
+
+-- !query
+/*
+ * various string functions
+ */
+select concat('one')
+-- !query schema
+struct<concat(one):string>
+-- !query output
+one
+
+
+-- !query
+select concat(1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat(1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: input to function concat should have been string, binary or array, but it's [int, int, int, string, boolean, boolean, date]; line 1 pos 7
+
+
+-- !query
+select concat_ws('#','one')
+-- !query schema
+struct<concat_ws(#, one):string>
+-- !query output
+one
+
+
+-- !query
+select concat_ws('#',1,2,3,'hello',true, false, to_date('20100309','yyyyMMdd'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('#', 1, 2, 3, 'hello', true, false, to_date('20100309', 'yyyyMMdd'))' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '1' is of int type. argument 3 requires (array<string> or string) type, however, '2' is of int type. argument 4 requires (array<string> or string) type, however, '3' is of int type. argument 6 requires (array<string> or string) type, however, 'true' is of boolean type. argument 7 requires (array<string> or string) type, however, 'false' is of boolean type. argument 8 requires (array<string> or string) type, however, 'to_date('20100309', 'yyyyMMdd')' is of date type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(',',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(',', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws('',10,20,null,30)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws('', 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select concat_ws(NULL,10,20,null,30) is null
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'concat_ws(CAST(NULL AS STRING), 10, 20, NULL, 30)' due to data type mismatch: argument 2 requires (array<string> or string) type, however, '10' is of int type. argument 3 requires (array<string> or string) type, however, '20' is of int type. argument 4 requires (array<string> or string) type, however, 'NULL' is of void type. argument 5 requires (array<string> or string) type, however, '30' is of int type.; line 1 pos 7
+
+
+-- !query
+select reverse('abcde')
+-- !query schema
+struct<reverse(abcde):string>
+-- !query output
+edcba
+
+
+-- !query
+select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+cannot resolve 'substring('ahoj', 1, t.i)' due to data type mismatch: argument 3 requires int type, however, 't.i' is of bigint type.; line 1 pos 10
+
+
+-- !query
+/*
+ * format
+ */
+select format_string(NULL)
+-- !query schema
+struct<format_string(NULL):string>
+-- !query output
+NULL
+
+
+-- !query
+select format_string('Hello')
+-- !query schema
+struct<format_string(Hello):string>
+-- !query output
+Hello
+
+
+-- !query
+select format_string('Hello %s', 'World')
+-- !query schema
+struct<format_string(Hello %s, World):string>
+-- !query output
+Hello World
+
+
+-- !query
+select format_string('Hello %%')
+-- !query schema
+struct<format_string(Hello %%):string>
+-- !query output
+Hello %
+
+
+-- !query
+select format_string('Hello %%%%')
+-- !query schema
+struct<format_string(Hello %%%%):string>
+-- !query output
+Hello %%
+
+
+-- !query
+select format_string('Hello %s %s', 'World')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %s')
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%s'
+
+
+-- !query
+select format_string('Hello %x', 20)
+-- !query schema
+struct<format_string(Hello %x, 20):string>
+-- !query output
+Hello 14
+
+
+-- !query
+select format_string('%1$s %3$s', 1, 2, 3)
+-- !query schema
+struct<format_string(%1$s %3$s, 1, 2, 3):string>
+-- !query output
+1 3
+
+
+-- !query
+select format_string('%1$s %12$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<format_string(%1$s %12$s, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12):string>
+-- !query output
+1 12
+
+
+-- !query
+select format_string('%1$s %4$s', 1, 2, 3)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%4$s'
+
+
+-- !query
+select format_string('%1$s %13$s', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
+-- !query schema
+struct<>
+-- !query output
+java.util.MissingFormatArgumentException
+Format specifier '%13$s'
+
+
+-- !query
+select format_string('%0$s', 'Hello')
+-- !query schema
+struct<>
+-- !query output
+java.util.IllegalFormatArgumentIndexException
+Illegal format argument index = 0
+

Review comment:
       @srowen @dongjoon-hyun Does this behavior need to be consistent with Java 8?
   
   https://github.com/apache/spark/blob/128168d8c4019a1e10a9f1be734868524f6a09f0/sql/core/src/test/resources/sql-tests/inputs/postgreSQL/text.sql#L90-L91
   
   It seems that Java 17 is same as PostgreSQL, but Java 8 returns the a non ERROR result 
   
   https://github.com/apache/spark/blob/128168d8c4019a1e10a9f1be734868524f6a09f0/sql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql.out#L274-L279
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939786068


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48547/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930770855


   **[Test build #143745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143745/testReport)** for PR 34153 at commit [`09fcb3e`](https://github.com/apache/spark/commit/09fcb3eda1797d1d8c039255c7c5032cf1be9ab2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r725794406



##########
File path: sql/core/pom.xml
##########
@@ -112,24 +112,6 @@
       <groupId>org.apache.parquet</groupId>
       <artifactId>parquet-column</artifactId>
     </dependency>
-    <dependency>

Review comment:
       Wrong merge, revert later
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-930828991


   **[Test build #143755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143755/testReport)** for PR 34153 at commit [`1761e74`](https://github.com/apache/spark/commit/1761e7474be5fdd73f9157b70f919f8ff0e0d535).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs with JDK 17 and Maven

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719074719



##########
File path: pom.xml
##########
@@ -2737,7 +2739,7 @@
             <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
             <junitxml>.</junitxml>
             <filereports>SparkTestSuite.txt</filereports>
-            <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+            <argLine>-ea -Xmx5g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>

Review comment:
       That sounds like a blocker for JDK17 adoption. Could you file a JIRA instead? Otherwise, this will bite us in the future.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931215600


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143763/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-938019873


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48473/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-939659783


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48543/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r725761934



##########
File path: pom.xml
##########
@@ -3431,6 +3433,26 @@
 
     </profile>
 
+    <profile>
+      <id>jdk-17</id>
+      <properties>
+        <extraJavaTestArgs>
+          --add-opens java.base/java.nio=ALL-UNNAMED

Review comment:
       @dongjoon-hyun  this profile already removed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #34153: [WIP][SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34153:
URL: https://github.com/apache/spark/pull/34153#discussion_r719124756



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##########
@@ -322,6 +323,13 @@ private object DateTimeFormatterHelper {
             (isParsing && unsupportedLettersForParsing.contains(c))) {
             throw new IllegalArgumentException(s"Illegal pattern character: $c")
           }
+          // SPARK-36796: `select date_format('2018-11-17 13:33:33.333', 'B')` failed with Java 8,
+          // but use Java 17 will return `in the afternoon` because 'B' is used to represent
+          // `Pattern letters to output a day period` in Java 17 and disabled it here for
+          // compatibility with Java 8 behavior.
+          for (c <- patternPart if unknownPatternLetters.contains(c)) {

Review comment:
       @dongjoon-hyun  @wangyum the result of  `select date_format('2018-11-17 13:33:33.333', 'B')` in `datetime-formatting-invalid.sql` with Java 8 is:
   ```
   -- !query
   select date_format('2018-11-17 13:33:33.333', 'B')
   -- !query schema
   struct<>
   -- !query output
   java.lang.IllegalArgumentException
   Unknown pattern letter: B
   ```
   
   With Java 17 the result is  
   ```
   - datetime-formatting-invalid.sql *** FAILED ***
     datetime-formatting-invalid.sql
     Expected "struct<[]>", but got "struct<[date_format(2018-11-17 13:33:33.333, B):string]>" Schema did not match for query #34
     select date_format('2018-11-17 13:33:33.333', 'B'): -- !query
     select date_format('2018-11-17 13:33:33.333', 'B')
     -- !query schema
     struct<date_format(2018-11-17 13:33:33.333, B):string>
     -- !query output
     in the afternoon (SQLQueryTestSuite.scala:469)
   ```
   
   'B' is used to represent `Pattern letters to output a day period` in Java 17 
   
   ```
   *  Pattern  Count  Equivalent builder methods
        *  -------  -----  --------------------------
        *    B       1      appendDayPeriodText(TextStyle.SHORT)
        *    BBBB    4      appendDayPeriodText(TextStyle.FULL)
        *    BBBBB   5      appendDayPeriodText(TextStyle.NARROW)
   ```
   
   Manual disabled it here for compatibility with Java 8 behavior.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #34153: [SPARK-36796][BUILD][CORE][SQL] Pass all `sql/core` and dependent modules UTs use Maven with JDK 17

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34153:
URL: https://github.com/apache/spark/pull/34153#issuecomment-931205281


   **[Test build #143762 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143762/testReport)** for PR 34153 at commit [`86517e6`](https://github.com/apache/spark/commit/86517e6f6bf4d88ed30cab78635c7c46d2f119c3).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org