You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "huaxingao (via GitHub)" <gi...@apache.org> on 2024/07/02 19:49:24 UTC

[PR] Initial Support for Spark 4.0 [iceberg]

huaxingao opened a new pull request, #10622:
URL: https://github.com/apache/iceberg/pull/10622

   This PR has the initial support for Spark 4.0. I use `v4.0.0-preview1` for now. Will switch to `v4.0.0`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao closed pull request #10622: WIP: Initial Support for Spark 4.0
URL: https://github.com/apache/iceberg/pull/10622


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10622:
URL: https://github.com/apache/iceberg/pull/10622#discussion_r1777339828


##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java:
##########
@@ -948,9 +948,9 @@ public void testAddFilesWithParallelism() {
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
-  private static final List<Object[]> EMPTY_QUERY_RESULT = Lists.newArrayList();
+  private static final List<Object[]> emptyQueryResult = Lists.newArrayList();

Review Comment:
   Not sure why we are changing the constant here



##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java:
##########
@@ -948,9 +948,9 @@ public void testAddFilesWithParallelism() {
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
-  private static final List<Object[]> EMPTY_QUERY_RESULT = Lists.newArrayList();
+  private static final List<Object[]> emptyQueryResult = Lists.newArrayList();
 
-  private static final StructField[] STRUCT = {
+  private static final StructField[] struct = {

Review Comment:
   or here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #10622:
URL: https://github.com/apache/iceberg/pull/10622#issuecomment-2233219459

   btw spark 4 is going to use parquet 1.14.1 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10622:
URL: https://github.com/apache/iceberg/pull/10622#discussion_r1777341211


##########
spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/RandomGeneratingUDF.java:
##########
@@ -29,7 +29,7 @@
 
 class RandomGeneratingUDF implements Serializable {
   private final long uniqueValues;
-  private final Random rand = new Random();

Review Comment:
   I don't think we should be having any changes for Spark3.5 in this PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #10622:
URL: https://github.com/apache/iceberg/pull/10622#issuecomment-2206939528

   Updated the description to link #10497, this is great to see @huaxingao 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #10622:
URL: https://github.com/apache/iceberg/pull/10622#issuecomment-2400850614

   cc @aihuaxu @RussellSpitzer 
   Spark 4.0 preview1 works OK now with  the new [PR](https://github.com/apache/iceberg/pull/11257), but there are still a few issues with Preview 2. I am working on fixing these new issues in Preview 2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10622:
URL: https://github.com/apache/iceberg/pull/10622#discussion_r1777326975


##########
.github/workflows/java-ci.yml:
##########
@@ -107,7 +107,7 @@ jobs:
     runs-on: ubuntu-22.04
     strategy:
       matrix:
-        jvm: [8, 11, 17]
+        jvm: [17]

Review Comment:
   This is a bit more of a serious change since we won't be supporting JVM 8 and 11 if we do this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10622:
URL: https://github.com/apache/iceberg/pull/10622#discussion_r1777338194


##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java:
##########
@@ -255,7 +255,7 @@ protected <T> void createOrReplaceView(String name, List<T> data, Encoder<T> enc
   private Dataset<Row> toDS(String schema, String jsonData) {
     List<String> jsonRows =
         Arrays.stream(jsonData.split("\n"))
-            .filter(str -> !str.trim().isEmpty())

Review Comment:
   Why this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #10622:
URL: https://github.com/apache/iceberg/pull/10622#issuecomment-2400841583

   It's easier to have a new PR than update this one. I am closing this one and open a new [PR](https://github.com/apache/iceberg/pull/11257)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org