You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Zoltan Chovan (Code Review)" <ge...@cloudera.org> on 2026/10/07 13:13:06 UTC
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Zoltan Chovan has uploaded this change for review. ( http://gerrit.cloudera.org:8080/25044
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
[spark] Migrate connector to Spark 4.1 + Scala 2.13
This change introduces Cross-build the Scala modules for Spark 4.1
(Scala 2.13) alongside the existing Spark 3.5 variant, which is also
moved to Scala 2.13.
The Spark 2 build profile was dropped. A -Pspark4 property selects
the Spark 4.1 build, the default remains Spark 3.5.
Both variants now build against Scala 2.13.
Scala 2.13 source fixes:
- parallel-collections module for kudu-backup's .par usage
- mutable-Buffer-vs-immutable-Seq (.toSeq)
- Array.deep removal and the mapValues view migrate
scala.collection.JavaConverters to scala.jdk.CollectionConverters
throughout
The one real Spark 3.5<->4.1 API delta (MemoryStream moved to the
.runtime package) is isolated behind a per-variant compat shim under
src/test/scala-spark{3,4}. DataSource stays on V1 (the V1 relations
and streaming Sink still exist in 4.1).
Test fixes surfaced by running the cluster-backed suite on both variants:
- Row.getAs[Seq[T]] on array columns returns a mutable.ArraySeq, but under
Scala 2.13 Seq aliases immutable.Seq, so the cast threw. Read the array
columns as scala.collection.Seq instead (KuduContextTest, SparkSQLTest).
- kuduRelationFromDataFrame extracted the relation from the raw logical
plan, but Spark 4's DataFrameReader no longer eagerly resolves the
source, so queryExecution.logical is an UnresolvedDataSource node. Use
the analyzed plan and match the LogicalRelation node type instead.
- Bind the driver to 127.0.0.1 in the test SparkConf: Spark 4 loads
whole-stage-codegen classes through the executor class loader, which
fetches from the driver's file server even in local[*] mode, so an
unreachable driver host makes every codegen query time out.
Manual testing: both variants build main+test and the full :kudu-spark:test
suite passes on Spark 3.5 and Spark 4.1.
The connector needs no ANSI-mode opt-out under Spark 4's default
spark.sql.ansi.enabled=true.
Change-Id: I040c1573b38a118183720281904e6956ff3672a6
---
M java/gradle/dependencies.gradle
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupGraph.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupIO.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/build.gradle
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupRDD.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-spark-tools/src/main/scala/org/apache/kudu/spark/tools/DistributedDataGenerator.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/ITBigLinkedListTest.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/TestImportExportFiles.scala
M java/kudu-spark/build.gradle
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/HdrHistogramAccumulator.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
A java/kudu-spark/src/test/scala-spark3/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
A java/kudu-spark/src/test/scala-spark4/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduRDDTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/StreamingTest.scala
27 files changed, 207 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/44/25044/1
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 5: Code-Review+1
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Nov 2026 13:44:17 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 2: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3129/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 10:19:47 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Gabriella Lotz, Attila Bukor, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25044
to look at the new patch set (#3).
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
[spark] Migrate connector to Spark 4.1 + Scala 2.13
This change introduces Cross-build the Scala modules for Spark 4.1
(Scala 2.13) alongside the existing Spark 3.5 variant, which is also
moved to Scala 2.13.
The Spark 2 build profile was dropped. A -Pspark4 property selects
the Spark 4.1 build, the default remains Spark 3.5.
Both variants now build against Scala 2.13.
Scala 2.13 source fixes:
- parallel-collections module for kudu-backup's .par usage
- mutable-Buffer-vs-immutable-Seq (.toSeq)
- Array.deep removal and the mapValues view migrate
scala.collection.JavaConverters to scala.jdk.CollectionConverters
throughout
The one real Spark 3.5<->4.1 API delta (MemoryStream moved to the
.runtime package) is isolated behind a per-variant compat shim under
src/test/scala-spark{3,4}. DataSource stays on V1 (the V1 relations
and streaming Sink still exist in 4.1).
Test fixes surfaced by running the cluster-backed suite on both variants:
- Row.getAs[Seq[T]] on array columns returns a mutable.ArraySeq, but under
Scala 2.13 Seq aliases immutable.Seq, so the cast threw. Read the array
columns as scala.collection.Seq instead (KuduContextTest, SparkSQLTest).
- kuduRelationFromDataFrame extracted the relation from the raw logical
plan, but Spark 4's DataFrameReader no longer eagerly resolves the
source, so queryExecution.logical is an UnresolvedDataSource node. Use
the analyzed plan and match the LogicalRelation node type instead.
- Bind the driver to 127.0.0.1 in the test SparkConf: Spark 4 loads
whole-stage-codegen classes through the executor class loader, which
fetches from the driver's file server even in local[*] mode, so an
unreachable driver host makes every codegen query time out.
Manual testing: both variants build main+test and the full :kudu-spark:test
suite passes on Spark 3.5 and Spark 4.1.
The connector needs no ANSI-mode opt-out under Spark 4's default
spark.sql.ansi.enabled=true.
Change-Id: I040c1573b38a118183720281904e6956ff3672a6
---
M java/config/spotbugs/excludeFilter.xml
M java/gradle/dependencies.gradle
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupGraph.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupIO.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/build.gradle
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupRDD.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-spark-tools/src/main/scala/org/apache/kudu/spark/tools/DistributedDataGenerator.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/ITBigLinkedListTest.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/TestImportExportFiles.scala
M java/kudu-spark/build.gradle
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/HdrHistogramAccumulator.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
A java/kudu-spark/src/test/scala-spark3/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
A java/kudu-spark/src/test/scala-spark4/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduRDDTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/StreamingTest.scala
28 files changed, 210 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/44/25044/3
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 3:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3140/
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Oct 2026 08:07:49 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Zoltan Chovan has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 5:
(1 comment)
https://gerrit.cloudera.org/#/c/25044/5/java/gradle/dependencies.gradle
File java/gradle/dependencies.gradle:
https://gerrit.cloudera.org/#/c/25044/5/java/gradle/dependencies.gradle@67
PS5, Line 67: spark
> What would be the issues arising if we name this to spark3 to maintain unif
Renaming this to spark3 would mean that we would need to add a condition for each line in libs[] where spark version needs to be selected. versions["spark"] can either be a spark3 or a spark4 version, it is determined the same way you are describing at line #80.
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Nov 2026 08:41:08 +0000
Gerrit-HasComments: Yes
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 6:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3379/
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Nov 2026 17:06:24 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 5: Verified+1
Build Successful
http://jenkins.kudu.apache.org/job/pre_commit/3238/ : SUCCESS
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Nov 2026 17:43:45 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Zoltan Chovan has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 5:
> Patch Set 5:
>
> (1 comment)
>
> I'm assuming there will be follow up changes to update the docs and build scripts to use spark4, right?
Yes docs need to be updated, but I'm not sure which build scripts are you referring to? The gradle scripts are already updated.
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Nov 2026 08:42:03 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 5:
(1 comment)
I'm assuming there will be follow up changes to update the docs and build scripts to use spark4, right?
https://gerrit.cloudera.org/#/c/25044/5/java/gradle/dependencies.gradle
File java/gradle/dependencies.gradle:
https://gerrit.cloudera.org/#/c/25044/5/java/gradle/dependencies.gradle@67
PS5, Line 67: spark
What would be the issues arising if we name this to spark3 to maintain uniformity in naming?
I guess below, we could do something like:
If the user passes spark3 or doesn't pass any spark property(default), we could use spark3. If spark4 is passed, we could use spark4.
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Nov 2026 01:07:22 +0000
Gerrit-HasComments: Yes
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Zoltan Chovan has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 5:
(5 comments)
https://gerrit.cloudera.org/#/c/25044/3/java/gradle/dependencies.gradle
File java/gradle/dependencies.gradle:
https://gerrit.cloudera.org/#/c/25044/3/java/gradle/dependencies.gradle@61
PS3, Line 61: // Match the scala-parallel-collections version shipped by each Spark variant.
> Spark 3.5 (Scala 2.13) ships scala-parallel-collections 1.0.4, but we compi
Done. scalaParallelCollections now defaults to 1.0.4, which is what Spark 3.5 ships. A new scalaParallelCollections4 (1.2.0) replaces it when building with -Pspark4, the same way spark4 overrides the Spark version.
https://gerrit.cloudera.org/#/c/25044/3/java/kudu-backup/build.gradle
File java/kudu-backup/build.gradle:
https://gerrit.cloudera.org/#/c/25044/3/java/kudu-backup/build.gradle@55
PS3, Line 55: // module; the Scala 2.13 Spark runtime provides it, so it is compile-only like scala-library.
> Nit: 'the Spark 2.13 runtime' -> 'the Scala 2.13 Spark runtime'.
Done
https://gerrit.cloudera.org/#/c/25044/3/java/kudu-backup/build.gradle@77
PS3, Line 77: testImplementation(libs.sparkAvro) {
> This forces 1.2.0 onto the Spark 3.5 test classpath, so tests don't run aga
Good point, it isn't needed. I removed the testImplementation line, so the test classpath now gets the version spark-core brings in transitively. The compileOnlyApi dependency is still there for compiling the .par usages.
https://gerrit.cloudera.org/#/c/25044/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala:
https://gerrit.cloudera.org/#/c/25044/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala@75
PS3, Line 75: writeArray(
> In 2.13 this copies into a List, and writeArray() indexes it with seq(i) in
Done. writeArray() now takes a scala.collection.Seq[Any], and the call site reads the column with row.getAs[scala.collection.Seq[Any]](sparkIdx) instead of getList(...).asScala.toSeq. Spark gives us a mutable.ArraySeq for array columns, so seq(i) is O(1) again and nothing gets copied. The CollectionConverters import was no longer used, so I dropped it.
https://gerrit.cloudera.org/#/c/25044/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
File java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala:
https://gerrit.cloudera.org/#/c/25044/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala@909
PS3, Line 909: while (System.currentTimeMillis() <= timestamp) {
> Instead of sleeping, could we read the propagated timestamp after the secon
Done. I replaced the sleep and added the assert you suggested: after the second batch, getLastPropagatedTimestampMs() > timestamp. On its own the assert would only turn a same-millisecond commit into a test failure, so before inserting the second batch the test now waits until System.currentTimeMillis() is past the snapshot boundary. That works because the mini cluster shares the host clock. With both in place the ordering is guaranteed and checked, instead of hoping 50 ms is enough.
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Nov 2026 17:02:23 +0000
Gerrit-HasComments: Yes
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 5:
(1 comment)
https://gerrit.cloudera.org/#/c/25044/5/java/gradle/dependencies.gradle
File java/gradle/dependencies.gradle:
https://gerrit.cloudera.org/#/c/25044/5/java/gradle/dependencies.gradle@67
PS5, Line 67: spark
> Renaming this to spark3 would mean that we would need to add a condition for each line in libs[] where spark version needs to be selected.
IIUC, the libs entries read $versions.spark and not $versions.spark3. If we set versions["spark"] once (in the if block in #80), before libs is built, none of the four spark* lines change. Something like this:
if (propertyExists("spark3") && propertyExists("spark4")) {
throw new GradleException("Only one of -Pspark3 or -Pspark4 may be specified")
}
def sparkVariant = propertyExists("spark4") ? "4" : "3"
versions["spark"] = versions["spark$sparkVariant"]
versions["scalaParallelCollections"] = versions["scalaParallelCollections$sparkVariant"]
Probably rename scalaParallelCollections to scalaParallelCollections3 in #62 to keep it consistent with the spark version.
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Nov 2026 18:33:23 +0000
Gerrit-HasComments: Yes
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Gabriella Lotz, Attila Bukor, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
https://gerrit.cloudera.org/25044
to look at the new patch set (#6).
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
[spark] Migrate connector to Spark 4.1 + Scala 2.13
This change introduces Cross-build the Scala modules for Spark 4.1
(Scala 2.13) alongside the existing Spark 3.5 variant, which is also
moved to Scala 2.13.
The Spark 2 build profile was dropped. A -Pspark4 property selects
the Spark 4.1 build, the default remains Spark 3.5.
Both variants now build against Scala 2.13.
Scala 2.13 source fixes:
- parallel-collections module for kudu-backup's .par usage
- mutable-Buffer-vs-immutable-Seq (.toSeq)
- Array.deep removal and the mapValues view migrate
scala.collection.JavaConverters to scala.jdk.CollectionConverters
throughout
The one real Spark 3.5<->4.1 API delta (MemoryStream moved to the
.runtime package) is isolated behind a per-variant compat shim under
src/test/scala-spark{3,4}. DataSource stays on V1 (the V1 relations
and streaming Sink still exist in 4.1).
Test fixes surfaced by running the cluster-backed suite on both variants:
- Row.getAs[Seq[T]] on array columns returns a mutable.ArraySeq, but under
Scala 2.13 Seq aliases immutable.Seq, so the cast threw. Read the array
columns as scala.collection.Seq instead (KuduContextTest, SparkSQLTest).
- kuduRelationFromDataFrame extracted the relation from the raw logical
plan, but Spark 4's DataFrameReader no longer eagerly resolves the
source, so queryExecution.logical is an UnresolvedDataSource node. Use
the analyzed plan and match the LogicalRelation node type instead.
- Bind the driver to 127.0.0.1 in the test SparkConf: Spark 4 loads
whole-stage-codegen classes through the executor class loader, which
fetches from the driver's file server even in local[*] mode, so an
unreachable driver host makes every codegen query time out.
Manual testing: both variants build main+test and the full :kudu-spark:test
suite passes on Spark 3.5 and Spark 4.1.
The connector needs no ANSI-mode opt-out under Spark 4's default
spark.sql.ansi.enabled=true.
Change-Id: I040c1573b38a118183720281904e6956ff3672a6
---
M java/config/spotbugs/excludeFilter.xml
M java/gradle/dependencies.gradle
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupGraph.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupIO.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/build.gradle
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupRDD.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-spark-tools/src/main/scala/org/apache/kudu/spark/tools/DistributedDataGenerator.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/ITBigLinkedListTest.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/TestImportExportFiles.scala
M java/kudu-spark/build.gradle
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/HdrHistogramAccumulator.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
A java/kudu-spark/src/test/scala-spark3/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
A java/kudu-spark/src/test/scala-spark4/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduRDDTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/StreamingTest.scala
28 files changed, 224 insertions(+), 77 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/44/25044/6
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 3: Code-Review+1
(5 comments)
http://gerrit.cloudera.org:8080/#/c/25044/3/java/gradle/dependencies.gradle
File java/gradle/dependencies.gradle:
http://gerrit.cloudera.org:8080/#/c/25044/3/java/gradle/dependencies.gradle@61
PS3, Line 61: scalaParallelCollections : "1.2.0",
Spark 3.5 (Scala 2.13) ships scala-parallel-collections 1.0.4, but we compile against 1.2.0. Should we pin the version per Spark variant so it matches the runtime?
http://gerrit.cloudera.org:8080/#/c/25044/3/java/kudu-backup/build.gradle
File java/kudu-backup/build.gradle:
http://gerrit.cloudera.org:8080/#/c/25044/3/java/kudu-backup/build.gradle@55
PS3, Line 55: // module; the Spark 2.13 runtime provides it, so it is compile-only like scala-library.
Nit: 'the Spark 2.13 runtime' -> 'the Scala 2.13 Spark runtime'.
http://gerrit.cloudera.org:8080/#/c/25044/3/java/kudu-backup/build.gradle@77
PS3, Line 77: testImplementation libs.scalaParallelCollections
This forces 1.2.0 onto the Spark 3.5 test classpath, so tests don't run against what Spark 3.5 provides at runtime. Is the explicit test dependency needed at all, given spark-core already brings it in transitively?
http://gerrit.cloudera.org:8080/#/c/25044/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala:
http://gerrit.cloudera.org:8080/#/c/25044/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala@75
PS3, Line 75: val seq = row.getList[Any](sparkIdx).asScala.toSeq
In 2.13 this copies into a List, and writeArray() indexes it with seq(i) in a loop, which is O(n^2) per cell. Could writeArray() take scala.collection.Seq (or IndexedSeq) and drop the .toSeq here?
http://gerrit.cloudera.org:8080/#/c/25044/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
File java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala:
http://gerrit.cloudera.org:8080/#/c/25044/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala@909
PS3, Line 909: Thread.sleep(50)
Instead of sleeping, could we read the propagated timestamp after the second batch and assert it is > timestamp? That would make the ordering explicit rather than timing-dependent.
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Oct 2026 14:17:11 +0000
Gerrit-HasComments: Yes
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 5:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3238/
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Nov 2026 16:59:23 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 2:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3129/
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Oct 2026 09:30:32 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Zoltan Chovan has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 6:
(1 comment)
https://gerrit.cloudera.org/#/c/25044/5/java/gradle/dependencies.gradle
File java/gradle/dependencies.gradle:
https://gerrit.cloudera.org/#/c/25044/5/java/gradle/dependencies.gradle@67
PS5, Line 67: spark
> > Renaming this to spark3 would mean that we would need to add a condition
Done
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Nov 2026 17:06:24 +0000
Gerrit-HasComments: Yes
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3087/
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 07 Oct 2026 13:13:12 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Gabriella Lotz, Attila Bukor, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25044
to look at the new patch set (#2).
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
[spark] Migrate connector to Spark 4.1 + Scala 2.13
This change introduces Cross-build the Scala modules for Spark 4.1
(Scala 2.13) alongside the existing Spark 3.5 variant, which is also
moved to Scala 2.13.
The Spark 2 build profile was dropped. A -Pspark4 property selects
the Spark 4.1 build, the default remains Spark 3.5.
Both variants now build against Scala 2.13.
Scala 2.13 source fixes:
- parallel-collections module for kudu-backup's .par usage
- mutable-Buffer-vs-immutable-Seq (.toSeq)
- Array.deep removal and the mapValues view migrate
scala.collection.JavaConverters to scala.jdk.CollectionConverters
throughout
The one real Spark 3.5<->4.1 API delta (MemoryStream moved to the
.runtime package) is isolated behind a per-variant compat shim under
src/test/scala-spark{3,4}. DataSource stays on V1 (the V1 relations
and streaming Sink still exist in 4.1).
Test fixes surfaced by running the cluster-backed suite on both variants:
- Row.getAs[Seq[T]] on array columns returns a mutable.ArraySeq, but under
Scala 2.13 Seq aliases immutable.Seq, so the cast threw. Read the array
columns as scala.collection.Seq instead (KuduContextTest, SparkSQLTest).
- kuduRelationFromDataFrame extracted the relation from the raw logical
plan, but Spark 4's DataFrameReader no longer eagerly resolves the
source, so queryExecution.logical is an UnresolvedDataSource node. Use
the analyzed plan and match the LogicalRelation node type instead.
- Bind the driver to 127.0.0.1 in the test SparkConf: Spark 4 loads
whole-stage-codegen classes through the executor class loader, which
fetches from the driver's file server even in local[*] mode, so an
unreachable driver host makes every codegen query time out.
Manual testing: both variants build main+test and the full :kudu-spark:test
suite passes on Spark 3.5 and Spark 4.1.
The connector needs no ANSI-mode opt-out under Spark 4's default
spark.sql.ansi.enabled=true.
Change-Id: I040c1573b38a118183720281904e6956ff3672a6
---
M java/config/spotbugs/excludeFilter.xml
M java/gradle/dependencies.gradle
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupGraph.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupIO.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/build.gradle
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupRDD.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-spark-tools/src/main/scala/org/apache/kudu/spark/tools/DistributedDataGenerator.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/ITBigLinkedListTest.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/TestImportExportFiles.scala
M java/kudu-spark/build.gradle
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/HdrHistogramAccumulator.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
A java/kudu-spark/src/test/scala-spark3/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
A java/kudu-spark/src/test/scala-spark4/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduRDDTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/StreamingTest.scala
28 files changed, 210 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/44/25044/2
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 3: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3140/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Oct 2026 08:47:17 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 5: Code-Review+2
(1 comment)
https://gerrit.cloudera.org/#/c/25044/5//COMMIT_MSG
Commit Message:
https://gerrit.cloudera.org/#/c/25044/5//COMMIT_MSG@41
PS5, Line 41: Manual testing: both variants build main+test and the full :kudu-spark:test
: suite passes on Spark 3.5 and Spark 4.1.
q: did you also run :kudu-backup:test and :kudu-spark-tools:test?
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Nov 2026 14:39:50 +0000
Gerrit-HasComments: Yes
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 1: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3087/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/25044
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 13:21:32 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Gabriella Lotz, Attila Bukor, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
https://gerrit.cloudera.org/25044
to look at the new patch set (#4).
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
[spark] Migrate connector to Spark 4.1 + Scala 2.13
This change introduces Cross-build the Scala modules for Spark 4.1
(Scala 2.13) alongside the existing Spark 3.5 variant, which is also
moved to Scala 2.13.
The Spark 2 build profile was dropped. A -Pspark4 property selects
the Spark 4.1 build, the default remains Spark 3.5.
Both variants now build against Scala 2.13.
Scala 2.13 source fixes:
- parallel-collections module for kudu-backup's .par usage
- mutable-Buffer-vs-immutable-Seq (.toSeq)
- Array.deep removal and the mapValues view migrate
scala.collection.JavaConverters to scala.jdk.CollectionConverters
throughout
The one real Spark 3.5<->4.1 API delta (MemoryStream moved to the
.runtime package) is isolated behind a per-variant compat shim under
src/test/scala-spark{3,4}. DataSource stays on V1 (the V1 relations
and streaming Sink still exist in 4.1).
Test fixes surfaced by running the cluster-backed suite on both variants:
- Row.getAs[Seq[T]] on array columns returns a mutable.ArraySeq, but under
Scala 2.13 Seq aliases immutable.Seq, so the cast threw. Read the array
columns as scala.collection.Seq instead (KuduContextTest, SparkSQLTest).
- kuduRelationFromDataFrame extracted the relation from the raw logical
plan, but Spark 4's DataFrameReader no longer eagerly resolves the
source, so queryExecution.logical is an UnresolvedDataSource node. Use
the analyzed plan and match the LogicalRelation node type instead.
- Bind the driver to 127.0.0.1 in the test SparkConf: Spark 4 loads
whole-stage-codegen classes through the executor class loader, which
fetches from the driver's file server even in local[*] mode, so an
unreachable driver host makes every codegen query time out.
Manual testing: both variants build main+test and the full :kudu-spark:test
suite passes on Spark 3.5 and Spark 4.1.
The connector needs no ANSI-mode opt-out under Spark 4's default
spark.sql.ansi.enabled=true.
Change-Id: I040c1573b38a118183720281904e6956ff3672a6
---
M java/config/spotbugs/excludeFilter.xml
M java/gradle/dependencies.gradle
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupGraph.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/BackupIO.scala
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/build.gradle
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupRDD.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-spark-tools/src/main/scala/org/apache/kudu/spark/tools/DistributedDataGenerator.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/ITBigLinkedListTest.scala
M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/TestImportExportFiles.scala
M java/kudu-spark/build.gradle
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/HdrHistogramAccumulator.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
A java/kudu-spark/src/test/scala-spark3/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
A java/kudu-spark/src/test/scala-spark4/org/apache/kudu/spark/kudu/MemoryStreamCompat.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduRDDTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/StreamingTest.scala
28 files changed, 221 insertions(+), 76 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/44/25044/4
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 4:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/3237/ : ABORTED
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Nov 2026 16:59:42 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 4:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3237/
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Nov 2026 16:59:08 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] Migrate connector to Spark 4.1 + Scala 2.13
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( https://gerrit.cloudera.org/25044 )
Change subject: [spark] Migrate connector to Spark 4.1 + Scala 2.13
......................................................................
Patch Set 6: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3379/ : FAILURE
--
To view, visit https://gerrit.cloudera.org/25044
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I040c1573b38a118183720281904e6956ff3672a6
Gerrit-Change-Number: 25044
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Nov 2026 18:00:49 +0000
Gerrit-HasComments: No