You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org> on 2025/10/20 23:10:29 UTC
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Abhishek Chennaka has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23565
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
KUDU-1261 [Java] Add spark bindings for Array columns
Introduces end-to-end support for Array types in Kudu Spark bindings.
Extends RowConverter, SparkUtil, and KuduRDD to serialize and deserialize
array columns between Kudu and Spark SQL. Adds integration tests covering
read/write and round-trip conversion for primitive and string array types.
Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
6 files changed, 496 insertions(+), 35 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/23565/1
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23565
to look at the new patch set (#10).
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
KUDU-1261 [Java] Add spark bindings for Array columns
Introduces end-to-end support for Array types in Kudu Spark bindings.
Extends RowConverter, SparkUtil, and KuduRDD to serialize and deserialize
array columns between Kudu and Spark SQL. Adds integration tests covering
read/write and round-trip conversion for primitive and string array types.
Updates the kudu-backup metadata structure to accommodate array type
column schema.
Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
---
M java/kudu-backup-common/src/main/protobuf/backup.proto
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
11 files changed, 778 insertions(+), 61 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/23565/10
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 10
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala:
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala@169
PS3, Line 169: if (rowResult.isNull(i)) {
: columns(i) = null
Is this really necessary given the code at line 174?
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala:
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala@60
PS3, Line 60: case Type.NESTED =>
: throw new IllegalArgumentException(
: "Type.NESTED should not be converted directly;" +
: " handle via col.isArray/col.getElementType.")
: case other =>
: throw new IllegalArgumentException(s"No support for Kudu type $other")
Why not to update this utility function to include what's done code at lines 105 -- 114? Is there any concern with updating the signature of this function to switch using a single 'col' parameter?
From the git history, I could see this was a private utility function, residing in KuduRelation before it was moved here.
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala@87
PS3, Line 87: case _ =>
: throw new IllegalArgumentException(s"No support for Spark SQL type $dt")
I suspect this could be thrown from the control paths involving writeOptions.handleSchemaDrift case in writeRows, but I might misunderstand what's going on there.
Does it make sense to add a test scenario with schema drift involving new array type column into the corresponding test suite, at least to confirm it's not throwing if you don't expect this to be thrown in such cases?
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
File java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala:
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala@138
PS3, Line 138: .nullable(true)
Maybe for at least for column, make it non-nullable to whether it's OK and doesn't blow up somewhere?
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
File java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala:
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala@553
PS3, Line 553: }
Could you also add coverage for empty arrays and null array cells as well, please?
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 21 Oct 2025 01:59:27 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 3: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2018/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 21 Oct 2025 01:02:28 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 12: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2068/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 12
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 07:42:28 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 10:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
File java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala:
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala@270
PS10, Line 270: col.hasNestedTypeDescriptor
> What if this part isn't true when (colType == Type.NESTED): should an erro
Done
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala@272
PS10, Line 272: if (nestedPB.hasArrayDescriptor) {
> What should be done in the 'else' case as of now, when only 1d-arrays are s
restructured it to throw exception similar to the above comment if there is no ArrayDescriptor.
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala:
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala@169
PS10, Line 169: // Array write helper
> nit: add a summary what this helper does (just the essence)?
Done
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala:
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala@a129
PS10, Line 129:
:
:
:
:
:
:
> nit: I see this comment has been removed at least in PS10, isn't it relevan
That should be there. Adding it back. Removed it accidentally.
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 10
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 05:25:11 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 10:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2065/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 10
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 01:46:02 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23565
to look at the new patch set (#13).
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
KUDU-1261 [Java] Add spark bindings for Array columns
Introduces end-to-end support for Array types in Kudu Spark bindings.
Extends RowConverter, SparkUtil, and KuduRDD to serialize and deserialize
array columns between Kudu and Spark SQL. Adds integration tests covering
read/write and round-trip conversion for primitive and string array types.
Updates the kudu-backup metadata structure to accommodate array type
column schema.
Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
---
M java/kudu-backup-common/src/main/protobuf/backup.proto
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
11 files changed, 789 insertions(+), 55 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/23565/13
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 13
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 14: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 14
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 18:16:22 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 8: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2057/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 8
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 23 Oct 2025 20:44:53 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 1: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2010/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 20 Oct 2025 23:22:11 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 7:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/2054/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 7
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 23 Oct 2025 17:32:46 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 7:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2054/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 7
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 23 Oct 2025 17:28:49 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23565
to look at the new patch set (#8).
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
KUDU-1261 [Java] Add spark bindings for Array columns
Introduces end-to-end support for Array types in Kudu Spark bindings.
Extends RowConverter, SparkUtil, and KuduRDD to serialize and deserialize
array columns between Kudu and Spark SQL. Adds integration tests covering
read/write and round-trip conversion for primitive and string array types.
Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
7 files changed, 686 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/23565/8
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 8
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 12: Verified+1
unrelated test failures in known flaky tests (TSAN): org.apache.kudu.client.TestKuduClient, org.apache.kudu.spark.kudu.DefaultSourceTest
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 12
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 15:14:38 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 2:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/2012/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 20 Oct 2025 23:55:08 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 12:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2068/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 12
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 05:43:38 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 11:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2067/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 11
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 05:43:08 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23565
to look at the new patch set (#11).
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
KUDU-1261 [Java] Add spark bindings for Array columns
Introduces end-to-end support for Array types in Kudu Spark bindings.
Extends RowConverter, SparkUtil, and KuduRDD to serialize and deserialize
array columns between Kudu and Spark SQL. Adds integration tests covering
read/write and round-trip conversion for primitive and string array types.
Updates the kudu-backup metadata structure to accommodate array type
column schema.
Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
---
M java/kudu-backup-common/src/main/protobuf/backup.proto
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
11 files changed, 790 insertions(+), 55 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/23565/11
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 11
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
File java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@1182
PS12, Line 1182: Objects.equal(before.getName, after.getName) &&
: Objects.equal(before.getType, after.getType) &&
: Objects.equal(before.isKey, after.isKey) &&
: Objects.equal(before.isNullable, after.isNullable) &&
: defaultValuesMatch(before.getDefaultValue, after.getDefaultValue) &&
: Objects.equal(before.getDesiredBlockSize, after.getDesiredBlockSize) &&
: Objects.equal(before.getEncoding, after.getEncoding) &&
: Objects
: .equal(before.getCompressionAlgorithm, after.getCompressionAlgorithm) &&
: Objects.equal(before.getTypeAttributes, after.getTypeAttributes) &&
: Objects.equal(before.getComment, after.getComment) &&
: Objects.equal(before.isArray, after.isArray)
Just curious: isn't it possible to use ColumnSchema.equal() here instead this re-implementation of the equal() method for ColumnSchema?
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 12
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 06:07:12 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 13:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
File java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala:
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala@272
PS12, Line 272: IllegalArgumentExcept
> nit: would IllegalArgumentException be a better choice? From where does th
Done
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala@277
PS12, Line 277: IllegalArgumentExcept
> Also, IllegalArgumentException might be even a better choice than Unsupport
Done
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
File java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@1182
PS12, Line 1182: Objects.equal(before.getName, after.getName) &&
: Objects.equal(before.getType, after.getType) &&
: Objects.equal(before.isKey, after.isKey) &&
: Objects.equal(before.isNullable, after.isNullable) &&
: defaultValuesMatch(before.getDefaultValue, after.getDefaultValue) &&
: Objects.equal(before.getDesiredBlockSize, after.getDesiredBlockSize) &&
: Objects.equal(before.getEncoding, after.getEncoding) &&
: Objects
: .equal(before.getCompressionAlgorithm, after.getCompressionAlgorithm) &&
: Objects.equal(before.getTypeAttributes, after.getTypeAttributes) &&
: Objects.equal(before.getComment, after.getComment) &&
: Objects.equal(before.isArray, after.isArray)
> Just curious: isn't it possible to use ColumnSchema.equal() here instead th
The current implementation of ColumnSchema.equals() doesn't check all the fields. Not sure the background behind it, but that seems to be the reason.
return Objects.equals(name, that.name) &&
Objects.equals(type, that.type) &&
Objects.equals(key, that.key) &&
Objects.equals(keyUnique, that.keyUnique) &&
Objects.equals(autoIncrementing, that.autoIncrementing) &&
Objects.equals(typeAttributes, that.typeAttributes) &&
Objects.equals(comment, that.comment) &&
Objects.equals(nestedTypeDescriptor, that.nestedTypeDescriptor);
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 13
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 15:31:58 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 14: Verified+1
unrelated test failures:
* ASAN: KUDU-3598
* TSAN: org.apache.kudu.client.TestKuduClient (known flaky test)
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 14
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 18:16:17 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 14: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2070/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 14
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 18:10:42 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 8:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2057/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 8
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 23 Oct 2025 19:22:20 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23565
to look at the new patch set (#6).
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
KUDU-1261 [Java] Add spark bindings for Array columns
Introduces end-to-end support for Array types in Kudu Spark bindings.
Extends RowConverter, SparkUtil, and KuduRDD to serialize and deserialize
array columns between Kudu and Spark SQL. Adds integration tests covering
read/write and round-trip conversion for primitive and string array types.
Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
7 files changed, 686 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/23565/6
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23565
to look at the new patch set (#5).
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
KUDU-1261 [Java] Add spark bindings for Array columns
Introduces end-to-end support for Array types in Kudu Spark bindings.
Extends RowConverter, SparkUtil, and KuduRDD to serialize and deserialize
array columns between Kudu and Spark SQL. Adds integration tests covering
read/write and round-trip conversion for primitive and string array types.
Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
7 files changed, 686 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/23565/5
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 5
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2010/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 20 Oct 2025 23:10:34 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 13: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2069/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 13
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 16:48:24 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 5:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2034/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 5
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 22 Oct 2025 06:04:45 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
File java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala:
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala@277
PS12, Line 277: IllegalStateException
> nit: would UnsupportedOperationException be a better choice here?
Also, IllegalArgumentException might be even a better choice than UnsupportedOperationException and IllegalStateException
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 12
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 06:09:09 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 11:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/2067/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 11
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 05:43:16 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 12:
> (1 comment)
BTW, all those look like nits, so feel free to ignore.
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 12
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 06:49:00 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 13:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2069/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 13
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 15:32:00 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 13: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
File java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@1182
PS12, Line 1182: Objects.equal(before.getName, after.getName) &&
: Objects.equal(before.getType, after.getType) &&
: Objects.equal(before.isKey, after.isKey) &&
: Objects.equal(before.isNullable, after.isNullable) &&
: defaultValuesMatch(before.getDefaultValue, after.getDefaultValue) &&
: Objects.equal(before.getDesiredBlockSize, after.getDesiredBlockSize) &&
: Objects.equal(before.getEncoding, after.getEncoding) &&
: Objects
: .equal(before.getCompressionAlgorithm, after.getCompressionAlgorithm) &&
: Objects.equal(before.getTypeAttributes, after.getTypeAttributes) &&
: Objects.equal(before.getComment, after.getComment) &&
: Objects.equal(before.isArray, after.isArray)
> The current implementation of ColumnSchema.equals() doesn't check all the f
Thanks, I'll try to check with the git history then. I just wanted to understand why.
Anyway, I assume it's supposed to be like this as of this PS.
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 13
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 15:41:07 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 4: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2024/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 21 Oct 2025 21:00:42 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 14
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 2:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2012/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 20 Oct 2025 23:24:39 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 10: Code-Review+1
(4 comments)
I took a quick look, overall I didn't notice any issues, but take this with a grain of salt: since I'm not expert in Scala, I'm not sure I can reliably spot problems, even if they are present. Adding +1 for a while.
From my side, I'll try to focus on addressing the test failure -- it seems that's something related to modifying not-null bitset in ColumnDataView()
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
File java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala:
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala@270
PS10, Line 270: col.hasNestedTypeDescriptor
What if this part isn't true when (colType == Type.NESTED): should an error be reported in such cases?
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala@272
PS10, Line 272: if (nestedPB.hasArrayDescriptor) {
What should be done in the 'else' case as of now, when only 1d-arrays are supported?
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala:
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala@169
PS10, Line 169: // Array write helper
nit: add a summary what this helper does (just the essence)?
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala:
http://gerrit.cloudera.org:8080/#/c/23565/10/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala@a129
PS10, Line 129:
:
:
:
:
:
:
nit: I see this comment has been removed at least in PS10, isn't it relevant anymore?
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 10
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 03:44:35 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 14:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2070/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 14
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 16:54:45 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 12: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
File java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala:
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala@272
PS12, Line 272: IllegalStateException
nit: would IllegalArgumentException be a better choice? From where does the 'metadata' parameter come in here?
http://gerrit.cloudera.org:8080/#/c/23565/12/java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala@277
PS12, Line 277: IllegalStateException
nit: would UnsupportedOperationException be a better choice here?
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 12
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 06:00:31 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
KUDU-1261 [Java] Add spark bindings for Array columns
Introduces end-to-end support for Array types in Kudu Spark bindings.
Extends RowConverter, SparkUtil, and KuduRDD to serialize and deserialize
array columns between Kudu and Spark SQL. Adds integration tests covering
read/write and round-trip conversion for primitive and string array types.
Updates the kudu-backup metadata structure to accommodate array type
column schema.
Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Reviewed-on: http://gerrit.cloudera.org:8080/23565
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Alexey Serbin <al...@apache.org>
---
M java/kudu-backup-common/src/main/protobuf/backup.proto
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
11 files changed, 789 insertions(+), 55 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 15
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 6: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2036/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 22 Oct 2025 10:41:48 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 4:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2024/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 21 Oct 2025 20:10:16 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 10: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2065/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 10
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 03:29:15 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 9:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/2063/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 9
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 01:42:53 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 9:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2063/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 9
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Oct 2025 01:42:37 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 12
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23565
to look at the new patch set (#3).
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
KUDU-1261 [Java] Add spark bindings for Array columns
Introduces end-to-end support for Array types in Kudu Spark bindings.
Extends RowConverter, SparkUtil, and KuduRDD to serialize and deserialize
array columns between Kudu and Spark SQL. Adds integration tests covering
read/write and round-trip conversion for primitive and string array types.
Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/RowConverter.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/SparkSQLTest.scala
6 files changed, 496 insertions(+), 35 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/23565/3
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 3:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2018/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 20 Oct 2025 23:55:16 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 4:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala:
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala@169
PS3, Line 169: if (rowResult.isNull(i)) {
: columns(i) = null
> Is this really necessary given the code at line 174?
Not really, it's already covered in line 173. Removed it.
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala:
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala@60
PS3, Line 60: case Type.NESTED =>
: throw new IllegalArgumentException(
: "Type.NESTED should not be converted directly;" +
: " handle via col.isArray/col.getElementType.")
: case other =>
: throw new IllegalArgumentException(s"No support for Kudu type $other")
> Why not to update this utility function to include what's done code at line
Done
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/SparkUtil.scala@87
PS3, Line 87: case _ =>
: throw new IllegalArgumentException(s"No support for Spark SQL type $dt")
> I suspect this could be thrown from the control paths involving writeOption
Yes, that is good catch. I added the needed logic in KuduContext.scala along with the test needed.
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala
File java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala:
http://gerrit.cloudera.org:8080/#/c/23565/3/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduTestSuite.scala@138
PS3, Line 138: .nullable(true)
> Maybe for at least for column, make it non-nullable to whether it's OK and
Done
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 22 Oct 2025 06:00:38 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 5:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/2034/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 5
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 22 Oct 2025 06:05:49 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1261 [Java] Add spark bindings for Array columns
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23565 )
Change subject: KUDU-1261 [Java] Add spark bindings for Array columns
......................................................................
Patch Set 6:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2036/
--
To view, visit http://gerrit.cloudera.org:8080/23565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I786fdd1cbfbb67b4895b2e95b89addbc04341746
Gerrit-Change-Number: 23565
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 22 Oct 2025 06:05:54 +0000
Gerrit-HasComments: No