You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2026/10/07 12:24:42 UTC

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/25043


Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................

IMPALA-15506: Add utility to test Spark with impala minicluster

Add convinience scripts to test Spark (4.1.3 by default) with
the Impala minicluster. Based on similar scripts for Trino:
- create docker image with build-spark-docker-image.sh
- start/stop/run sql with run-spark.sh/kill-spark.sh/spark-sql.sh

No tests added at this point, to goal is only to help with testing
Spark<->Impala interop.

The main motivation was GEOMETRY testing, so newest (1.12.0) Iceberg
is used in Spark instead of Impala 1.11.0. Versions used can be
overriden with env vars when creating the docker image.

Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
---
A testdata/bin/SPARK-README.md
A testdata/bin/build-spark-docker-image.sh
A testdata/bin/kill-spark.sh
A testdata/bin/run-spark.sh
A testdata/bin/spark-sql.sh
5 files changed, 246 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/25043/1
-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/24388/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 16:00:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has removed a vote on this change.

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Removed Code-Review+2 by Zoltan Borok-Nagy <bo...@cloudera.com>
-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 3: Code-Review+2

Accidentally removed Zoltan's +2, nevertheless I have an open question.


-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 12:31:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/25043/2/testdata/bin/build-spark-docker-image.sh
File testdata/bin/build-spark-docker-image.sh:

http://gerrit.cloudera.org:8080/#/c/25043/2/testdata/bin/build-spark-docker-image.sh@64
PS2, Line 64: ${ICEBERG_VERSION}/${JAR_NAME}
What about mounting the JAR instead of rebuilding the spark image?



-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 12:30:06 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................

IMPALA-15506: Add utility to test Spark with impala minicluster

Add convenience scripts to test Spark (4.1.3 by default) with
the Impala minicluster. Based on similar scripts for Trino:
- create docker image with build-spark-docker-image.sh
- start/stop/run sql with run-spark.sh/kill-spark.sh/spark-sql.sh

No tests added at this point, to goal is only to help with testing
Spark<->Impala interop.

The main motivation was GEOMETRY testing, so newest (1.12.0) Iceberg
is used in Spark instead of Impala's 1.11.0. Versions used can be
overriden with env vars when creating the docker image.

Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
---
A testdata/bin/SPARK-README.md
A testdata/bin/build-spark-docker-image.sh
A testdata/bin/kill-spark.sh
A testdata/bin/run-spark.sh
A testdata/bin/spark-sql.sh
5 files changed, 246 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/25043/2
-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Peter Rozsa, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/25043

to look at the new patch set (#4).

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................

IMPALA-15506: Add utility to test Spark with impala minicluster

Add convenience scripts to test Spark (4.1.3 by default) with
the Impala minicluster. Based on similar scripts for Trino:
start/stop/run sql with run-spark.sh/kill-spark.sh/spark-sql.sh

The official Spark docker image is used unmodified, so there is no
image to build. run-spark.sh bind mounts the Iceberg runtime jar and
a conf directory assembled from the minicluster configuration, and
passes everything the image would otherwise bake in as environment
variables. The jar is downloaded from Maven Central on the first run.
Both mount sources live under testdata/bin/minicluster_spark and are
gitignored.

No tests added at this point, the goal is only to help with testing
Spark<->Impala interop.

The main motivation was GEOMETRY testing, so newest (1.12.0) Iceberg
is used in Spark instead of Impala's 1.11.0. Versions used can be
overridden with env vars read by run-spark.sh.

Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Assisted-by: Claude Opus 5
---
A testdata/bin/SPARK-README.md
A testdata/bin/kill-spark.sh
A testdata/bin/minicluster_spark/.gitignore
A testdata/bin/minicluster_spark/spark-defaults-iceberg.conf
A testdata/bin/minicluster_spark/spark-defaults.conf
A testdata/bin/run-spark.sh
A testdata/bin/spark-sql.sh
7 files changed, 383 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/25043/4
-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/25043/2/testdata/bin/build-spark-docker-image.sh
File testdata/bin/build-spark-docker-image.sh:

http://gerrit.cloudera.org:8080/#/c/25043/2/testdata/bin/build-spark-docker-image.sh@64
PS2, Line 64: 
> The commit message states that it will be used for testing, that implies th
done, removed this file and configuration is moved to the start script



-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 15:38:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 12:29:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 4: Code-Review+2

Thanks!


-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 15:43:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/25043

to look at the new patch set (#3).

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................

IMPALA-15506: Add utility to test Spark with impala minicluster

Add convenience scripts to test Spark (4.1.3 by default) with
the Impala minicluster. Based on similar scripts for Trino:
- create docker image with build-spark-docker-image.sh
- start/stop/run sql with run-spark.sh/kill-spark.sh/spark-sql.sh

No tests added at this point, to goal is only to help with testing
Spark<->Impala interop.

The main motivation was GEOMETRY testing, so newest (1.12.0) Iceberg
is used in Spark instead of Impala's 1.11.0. Versions used can be
overriden with env vars when creating the docker image.

Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Assisted-by: Claude Opus 5
---
A testdata/bin/SPARK-README.md
A testdata/bin/build-spark-docker-image.sh
A testdata/bin/kill-spark.sh
A testdata/bin/run-spark.sh
A testdata/bin/spark-sql.sh
5 files changed, 246 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/25043/3
-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/25043/2/testdata/bin/build-spark-docker-image.sh
File testdata/bin/build-spark-docker-image.sh:

http://gerrit.cloudera.org:8080/#/c/25043/2/testdata/bin/build-spark-docker-image.sh@64
PS2, Line 64: ${ICEBERG_VERSION}/${JAR_NAME}
> It makes sense to mount it, but for now I would prefer sticking to rebuildi
The commit message states that it will be used for testing, that implies that it will run on every GVO



-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 14:03:02 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 4: Verified+1 Code-Review+2

This patch doesn't change compiled code or tests, adding verified manually.


-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 16:59:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/25043/2/testdata/bin/build-spark-docker-image.sh
File testdata/bin/build-spark-docker-image.sh:

http://gerrit.cloudera.org:8080/#/c/25043/2/testdata/bin/build-spark-docker-image.sh@64
PS2, Line 64: ${ICEBERG_VERSION}/${JAR_NAME}
> What about mounting the JAR instead of rebuilding the spark image?
It makes sense to mount it, but for now I would prefer sticking to rebuilding. I don't think that it will be done too often.



-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 12:35:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/24383/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 12:45:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15506: Add utility to test Spark with impala minicluster

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/25043 )

Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................

IMPALA-15506: Add utility to test Spark with impala minicluster

Add convenience scripts to test Spark (4.1.3 by default) with
the Impala minicluster. Based on similar scripts for Trino:
start/stop/run sql with run-spark.sh/kill-spark.sh/spark-sql.sh

The official Spark docker image is used unmodified, so there is no
image to build. run-spark.sh bind mounts the Iceberg runtime jar and
a conf directory assembled from the minicluster configuration, and
passes everything the image would otherwise bake in as environment
variables. The jar is downloaded from Maven Central on the first run.
Both mount sources live under testdata/bin/minicluster_spark and are
gitignored.

No tests added at this point, the goal is only to help with testing
Spark<->Impala interop.

The main motivation was GEOMETRY testing, so newest (1.12.0) Iceberg
is used in Spark instead of Impala's 1.11.0. Versions used can be
overridden with env vars read by run-spark.sh.

Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Assisted-by: Claude Opus 5
Reviewed-on: http://gerrit.cloudera.org:8080/25043
Reviewed-by: Peter Rozsa <pr...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Csaba Ringhofer <cs...@cloudera.com>
---
A testdata/bin/SPARK-README.md
A testdata/bin/kill-spark.sh
A testdata/bin/minicluster_spark/.gitignore
A testdata/bin/minicluster_spark/spark-defaults-iceberg.conf
A testdata/bin/minicluster_spark/spark-defaults.conf
A testdata/bin/run-spark.sh
A testdata/bin/spark-sql.sh
7 files changed, 383 insertions(+), 0 deletions(-)

Approvals:
  Peter Rozsa: Looks good to me, approved
  Csaba Ringhofer: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>