You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2026/07/03 16:47:28 UTC
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24544
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Add a custom cluster test suite that exercises Impala <-> Trino
interoperability over Apache Iceberg V3 tables, plus the test-infra
support to drive and verify Trino from .test files.
Test infra:
- New TRINO_QUERY / TRINO_RESULTS .test sections, analogous to
HIVE_QUERY / RESULTS. TRINO_QUERY statements run against the Trino
container (iceberg catalog, test database as schema); TRINO_RESULTS
verifies the returned rows (with an optional VERIFY_* modifier).
- tests/common/trino_cluster.py: TrinoCluster lifecycle helper
(start/stop/is_container_running) that drives the Trino CLI inside the
'impala-minicluster-trino' container via 'docker exec', so no new
Python dependency is required. Results are formatted into the Impala
RESULTS textual convention and compared as opaque strings. start()
builds the image (build-trino-docker-image.sh) lazily if it is missing
and starts the container, so the Docker cost is paid only by runs that
actually execute the interop tests (mirrors the MySQL JDBC tests) and
never by data loading, cluster setup or run-all-tests.sh. start() is
guarded by an inter-process file lock and the container is not stopped
from an individual pytest-xdist worker, so the suite is safe to run
concurrently (e.g. -n3).
- run_test_case dispatches TRINO_QUERY to Trino and verifies against
TRINO_RESULTS; run_stmt_in_trino mirrors run_stmt_in_hive.
- Disable Trino's DELTA_LENGTH_BYTE_ARRAY Parquet encoding for the
Iceberg catalog, which Impala cannot read for string/binary columns
yet (Trino 482 defaults it on). This is applied server-side via the
image's file session property manager (session-property-config.*),
setting parquet_writer_delta_length_byte_array_encoding_enabled=false.
Tests (tests/custom_cluster/test_iceberg_trino_interop.py; run in the
core exploration strategy, skipped only on non-HDFS filesystems, and
failing rather than skipping if the Trino container/Docker is
unavailable so problems are visible):
- INSERT / SELECT round-trips (both directions), format-version check.
- DELETE / UPDATE / MERGE materialized as Puffin deletion vectors (both
directions), asserting the delete files are Puffin DVs via Iceberg
"$files" metadata.
- Column default values written by one engine and read by the other.
Pin the Trino Docker image to trinodb/trino:482 instead of 'latest' so
the interop tests are reproducible and not made flaky by future Trino
releases (482 is the first release with full Iceberg V3 support).
Docs: testdata/bin/TRINO-README.md explains how to run the suite and
author TRINO_QUERY / TRINO_RESULTS sections.
Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Assisted-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
M testdata/bin/TRINO-README.md
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/session-property-config.json
A testdata/bin/minicluster_trino/session-property-config.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-default-values.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-delete-dv.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-insert.test
M tests/common/impala_test_suite.py
M tests/common/test_result_verifier.py
A tests/common/trino_cluster.py
A tests/custom_cluster/test_iceberg_trino_interop.py
M tests/util/test_file_parser.py
12 files changed, 715 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/24544/1
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2026 12:35:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 3:
(6 comments)
Thanks or the comments!
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/common/test_result_verifier.py
File tests/common/test_result_verifier.py:
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/common/test_result_verifier.py@539
PS3, Line 539: if verifier and verifier.upper() == 'VERIFY_IS_EQUAL':
: order_matters = True
: if verifier and verifier.upper() == 'VERIFY_IS_EQUAL_SORTED':
: order_matters = False
> Can you add tests that use these?
Done
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py
File tests/custom_cluster/test_trino_interop.py:
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@59
PS3, Line 59: v
> flake8: U100 Unused argument 'vector'
Done
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@88
PS3, Line 88: unique_database
> The database created here is not used - could use unique_name fixture.
Done
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@88
PS3, Line 88: v
> flake8: U100 Unused argument 'vector'
Done
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@110
PS3, Line 110: )
> flake8: E501 line too long (92 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@113
PS3, Line 113: # Impala only learns about out-of-band HMS objects after an INVALIDATE.
: self.execute_query("INVALIDATE METADATA")
> HMS events still work, right? So Impala sees the database after bit of time
Switched to event-based check.
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Jul 2026 11:33:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2026 12:35:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 2:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24544/2/tests/custom_cluster/test_trino_interop.py
File tests/custom_cluster/test_trino_interop.py:
http://gerrit.cloudera.org:8080/#/c/24544/2/tests/custom_cluster/test_trino_interop.py@59
PS2, Line 59: v
flake8: U100 Unused argument 'vector'
http://gerrit.cloudera.org:8080/#/c/24544/2/tests/custom_cluster/test_trino_interop.py@88
PS2, Line 88: v
flake8: U100 Unused argument 'vector'
http://gerrit.cloudera.org:8080/#/c/24544/2/tests/custom_cluster/test_trino_interop.py@110
PS2, Line 110: )
flake8: E501 line too long (92 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2026 19:27:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 4: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Jul 2026 14:22:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Add a custom cluster test suite that exercises Impala <-> Trino
interoperability over Apache Iceberg V3 tables, plus the test-infra
support to drive and verify Trino from .test files.
Test infra:
- New TRINO_QUERY / RESULTS .test sections, analogous to
HIVE_QUERY / RESULTS. TRINO_QUERY statements run against the Trino
container (iceberg catalog, test database as schema); RESULTS
verifies the returned rows (with an optional VERIFY_* modifier).
- tests/common/trino_cluster.py: TrinoCluster lifecycle helper
(start/stop/is_container_running) that drives the Trino CLI inside the
'impala-minicluster-trino' container via 'docker exec', so no new
Python dependency is required. Results are formatted into the Impala
RESULTS textual convention and compared as opaque strings. start()
builds the image (build-trino-docker-image.sh) lazily if it is missing
and starts the container, so the Docker cost is paid only by runs that
actually execute the interop tests (mirrors the MySQL JDBC tests) and
never by data loading, cluster setup or run-all-tests.sh.
- run_test_case dispatches TRINO_QUERY to Trino and verifies against
RESULTS; run_stmt_in_trino mirrors run_stmt_in_hive.
- CustomClusterTestSuite.with_args(run_trino=True): the framework starts
the Trino container before the Impala cluster and stops it on teardown
(only if it started it), so interop suites no longer hand-roll
setup_class/teardown_class. If the Impala setup fails after Trino
started, Trino is stopped so a failed setup does not leak the
container.
- Disable Trino's DELTA_LENGTH_BYTE_ARRAY Parquet encoding for the
Iceberg catalog, which Impala cannot read for string/binary columns
yet (Trino 482 defaults it on). This is applied server-side via the
image's file session property manager (session-property-config.*),
setting parquet_writer_delta_length_byte_array_encoding_enabled=false.
- Set hive.non-managed-table-writes-enabled=true in the image's
hive.properties. Our HMS runs the default metadata transformer, which
converts the MANAGED tables Trino requests into EXTERNAL ones (Trino
does not advertise Hive ACID write capabilities), and Trino refuses to
write into existing external tables by default; enabling this lets the
hive-catalog interop test use plain CREATE TABLE + INSERT.
Tests (run in run-tests.py's serial phase like other custom cluster
tests, skipped only on non-HDFS filesystems, and failing rather than
skipping if the Trino container/Docker is unavailable so problems are
visible):
- tests/custom_cluster/test_iceberg_trino_interop.py (iceberg catalog):
- tests/custom_cluster/test_trino_interop.py (hive catalog): a minimal
suite over legacy (non-Iceberg) Hive tables
Pin the Trino Docker image to trinodb/trino:482 instead of 'latest' so
the interop tests are reproducible and not made flaky by future Trino
releases (482 is the first release with full Iceberg V3 support).
Docs: testdata/bin/TRINO-README.md explains how to run the suites and
author TRINO_QUERY / RESULTS sections.
Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Assisted-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
Reviewed-on: http://gerrit.cloudera.org:8080/24544
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M testdata/bin/TRINO-README.md
M testdata/bin/minicluster_trino/Dockerfile
M testdata/bin/minicluster_trino/hive.properties
A testdata/bin/minicluster_trino/session-property-config.json
A testdata/bin/minicluster_trino/session-property-config.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-default-values.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-delete-dv.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-insert.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/test_result_verifier.py
A tests/common/trino_cluster.py
A tests/custom_cluster/test_iceberg_trino_interop.py
A tests/custom_cluster/test_trino_interop.py
M tests/util/test_file_parser.py
15 files changed, 892 insertions(+), 11 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 2:
(7 comments)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/24544/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24544/1//COMMIT_MSG@14
PS1, Line 14: RESULTS .test
> Why is TRINO_RESULTS needed where HIVE_QUERY uses simply RESULTS
Done
http://gerrit.cloudera.org:8080/#/c/24544/1//COMMIT_MSG@26
PS1, Line 26: cluster setup or run-all-tests.sh. start() is
: guarded by an inter-process file lock and the container is not stopped
: from an individual pytest-xdist worker, so the suite is safe to run
: concurrently (e.g. -n3).
> A ticket could be created for allowing Trino during data loading, both to s
This was not the intention(/leftover TODO) of this CR, but filed IMPALA-15143. Please feel free to file any ticket for further improvements.
http://gerrit.cloudera.org:8080/#/c/24544/1//COMMIT_MSG@38
PS1, Line 38: TH_BYTE_ARRAY Parquet enco
> Does interop work with Hive tables too?
I refactored the code and created test_trino_interop.py, but kept test_iceberg_trino_interop.py as well. If it ever gets backported to an old version (which I doubt) the backporter can just omit test_iceberg_trino_interop.py.
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/test_result_verifier.py
File tests/common/test_result_verifier.py:
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/test_result_verifier.py@521
PS1, Line 521: Because
: Trino's SQL type names differ from Impala's, every column is compared as an
: opaque STRING (i.e. exact textual match); we do not cross-check column types.
> Can you create a ticket for these? I agree that this is not critical now, b
Filed IMPALA-15149
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/trino_cluster.py
File tests/common/trino_cluster.py:
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/trino_cluster.py@97
PS1, Line 97: def _build_image(self):
: """Build the Trino Docker image via build-trino-docker-image.sh. This is done
: lazily (only when the image is missing) so that non-interop workflows never
: pay the cost; layer caching makes subsequent builds cheap. Raises
: TrinoUnavailable on failure so callers can skip."""
: script = os.path.join(IMPALA_HOME, 'testdata/bin/build-trino-docker-image.sh')
> It is nice to avoid cost when we don't need this, but I think that it would
I'd rather do it later when these test setups prove to be stable. Maybe it could be covered in the context of IMPALA-15143.
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/trino_cluster.py@183
PS1, Line 183: """Run a single SQL statement through the containerized Trino CLI. Returns a
: (stdout, stderr, returncode) tuple; does NOT raise on query error so callers
: can decide how to report failures."""
: cmd = ['docker', 'exec', self.container, 'trino',
: '--server=' + self.server, '--output-format=' + output_format]
> An alternative would be to add https://github.com/trinodb/trino-python-clie
If it's not necessary I'd rather not add extra dependencies.
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/custom_cluster/test_iceberg_trino_interop.py
File tests/custom_cluster/test_iceberg_trino_interop.py:
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/custom_cluster/test_iceberg_trino_interop.py@43
PS1, Line 43: @SkipIf.not_hdfs
> It seems more comfortable to me to add this feature to CustomClusterTestSui
Good idea, done.
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2026 19:33:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/24544/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24544/1//COMMIT_MSG@14
PS1, Line 14: TRINO_RESULTS
Why is TRINO_RESULTS needed where HIVE_QUERY uses simply RESULTS
http://gerrit.cloudera.org:8080/#/c/24544/1//COMMIT_MSG@26
PS1, Line 26: cluster setup or run-all-tests.sh. start() is
: guarded by an inter-process file lock and the container is not stopped
: from an individual pytest-xdist worker, so the suite is safe to run
: concurrently (e.g. -n3).
A ticket could be created for allowing Trino during data loading, both to support things not yet in Hive and to possibly speed loading up compared to Hive.
http://gerrit.cloudera.org:8080/#/c/24544/1//COMMIT_MSG@38
PS1, Line 38: test_iceberg_trino_interop
Does interop work with Hive tables too?
I would prefer to focus this patch on adding the test tooling, and adding more complex Iceberg tests in another patch. This would make it easier to backport this patch if needed to older branches without Iceberg v3 support.
Ideally there would be a test_trino_interop.py that looks for basic operations:
- can Trino see databases/tables created by Impala? (e.g. from functional db)
- can Trino create and write databases/tables and is Impala able to read these?
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/test_result_verifier.py
File tests/common/test_result_verifier.py:
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/test_result_verifier.py@521
PS1, Line 521: Because
: Trino's SQL type names differ from Impala's, every column is compared as an
: opaque STRING (i.e. exact textual match); we do not cross-check column types.
Can you create a ticket for these? I agree that this is not critical now, but it would be nice to have typed result checking.
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/trino_cluster.py
File tests/common/trino_cluster.py:
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/trino_cluster.py@97
PS1, Line 97: def _build_image(self):
: """Build the Trino Docker image via build-trino-docker-image.sh. This is done
: lazily (only when the image is missing) so that non-interop workflows never
: pay the cost; layer caching makes subsequent builds cheap. Raises
: TrinoUnavailable on failure so callers can skip."""
: script = os.path.join(IMPALA_HOME, 'testdata/bin/build-trino-docker-image.sh')
It is nice to avoid cost when we don't need this, but I think that it would be better to move this to an earlier phase of Impala builds, for example before dataload. If tests use trino that are in the core test suite then we generally can't avoid the cost of building it, and doing it randomly at the first test is harder to debug than doing it at a fixed place. It could be skipped with similar conditions as in tests (@SkipIf.not_hdfs).
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/common/trino_cluster.py@183
PS1, Line 183: """Run a single SQL statement through the containerized Trino CLI. Returns a
: (stdout, stderr, returncode) tuple; does NOT raise on query error so callers
: can decide how to report failures."""
: cmd = ['docker', 'exec', self.container, 'trino',
: '--server=' + self.server, '--output-format=' + output_format]
An alternative would be to add https://github.com/trinodb/trino-python-client (https://pypi.org/project/trino/) to the dependencies. This is a similar python dbapi client as impyla.
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/custom_cluster/test_iceberg_trino_interop.py
File tests/custom_cluster/test_iceberg_trino_interop.py:
http://gerrit.cloudera.org:8080/#/c/24544/1/tests/custom_cluster/test_iceberg_trino_interop.py@43
PS1, Line 43: @CustomClusterTestSuite.with_args(cluster_size=3)
It seems more comfortable to me to add this feature to CustomClusterTestSuite, with_args could have a has_trino parameter taking care of the things below. For example I would also like to use Trino for geospatial interop tests, which may possible go to another test class.
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2026 07:43:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13731/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2026 12:35:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13696/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 16:48:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2026 17:02:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24544
to look at the new patch set (#2).
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Add a custom cluster test suite that exercises Impala <-> Trino
interoperability over Apache Iceberg V3 tables, plus the test-infra
support to drive and verify Trino from .test files.
Test infra:
- New TRINO_QUERY / RESULTS .test sections, analogous to
HIVE_QUERY / RESULTS. TRINO_QUERY statements run against the Trino
container (iceberg catalog, test database as schema); RESULTS
verifies the returned rows (with an optional VERIFY_* modifier).
- tests/common/trino_cluster.py: TrinoCluster lifecycle helper
(start/stop/is_container_running) that drives the Trino CLI inside the
'impala-minicluster-trino' container via 'docker exec', so no new
Python dependency is required. Results are formatted into the Impala
RESULTS textual convention and compared as opaque strings. start()
builds the image (build-trino-docker-image.sh) lazily if it is missing
and starts the container, so the Docker cost is paid only by runs that
actually execute the interop tests (mirrors the MySQL JDBC tests) and
never by data loading, cluster setup or run-all-tests.sh. start() is
guarded by an inter-process file lock and the container is not stopped
from an individual pytest-xdist worker, so the suite is safe to run
concurrently (e.g. -n3).
- run_test_case dispatches TRINO_QUERY to Trino and verifies against
RESULTS; run_stmt_in_trino mirrors run_stmt_in_hive.
- CustomClusterTestSuite.with_args(run_trino=True): the framework starts
the Trino container before the Impala cluster and stops it on teardown
(only if it started it, and never mid-run under pytest-xdist), so
interop suites no longer hand-roll setup_class/teardown_class. If the
Impala setup fails after Trino started, Trino is stopped so a failed
setup does not leak the container.
- Disable Trino's DELTA_LENGTH_BYTE_ARRAY Parquet encoding for the
Iceberg catalog, which Impala cannot read for string/binary columns
yet (Trino 482 defaults it on). This is applied server-side via the
image's file session property manager (session-property-config.*),
setting parquet_writer_delta_length_byte_array_encoding_enabled=false.
- Set hive.non-managed-table-writes-enabled=true in the image's
hive.properties. Our HMS runs the default metadata transformer, which
converts the MANAGED tables Trino requests into EXTERNAL ones (Trino
does not advertise Hive ACID write capabilities), and Trino refuses to
write into existing external tables by default; enabling this lets the
hive-catalog interop test use plain CREATE TABLE + INSERT.
Tests (run in the core exploration strategy, skipped only on non-HDFS
filesystems, and failing rather than skipping if the Trino
container/Docker is unavailable so problems are visible):
- tests/custom_cluster/test_iceberg_trino_interop.py (iceberg catalog):
- tests/custom_cluster/test_trino_interop.py (hive catalog): a minimal
suite over legacy (non-Iceberg) Hive tables
Pin the Trino Docker image to trinodb/trino:482 instead of 'latest' so
the interop tests are reproducible and not made flaky by future Trino
releases (482 is the first release with full Iceberg V3 support).
Docs: testdata/bin/TRINO-README.md explains how to run the suites and
author TRINO_QUERY / RESULTS sections.
Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Assisted-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
M testdata/bin/TRINO-README.md
M testdata/bin/minicluster_trino/Dockerfile
M testdata/bin/minicluster_trino/hive.properties
A testdata/bin/minicluster_trino/session-property-config.json
A testdata/bin/minicluster_trino/session-property-config.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-default-values.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-delete-dv.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-insert.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/test_result_verifier.py
A tests/common/trino_cluster.py
A tests/custom_cluster/test_iceberg_trino_interop.py
A tests/custom_cluster/test_trino_interop.py
M tests/util/test_file_parser.py
15 files changed, 880 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/24544/2
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 3:
(3 comments)
Looks good in general, just a few comments.
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/common/test_result_verifier.py
File tests/common/test_result_verifier.py:
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/common/test_result_verifier.py@539
PS3, Line 539: if verifier and verifier.upper() == 'VERIFY_IS_EQUAL':
: order_matters = True
: if verifier and verifier.upper() == 'VERIFY_IS_EQUAL_SORTED':
: order_matters = False
Can you add tests that use these?
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py
File tests/custom_cluster/test_trino_interop.py:
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@88
PS3, Line 88: unique_database
The database created here is not used - could use unique_name fixture.
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@113
PS3, Line 113: # Impala only learns about out-of-band HMS objects after an INVALIDATE.
: self.execute_query("INVALIDATE METADATA")
HMS events still work, right? So Impala sees the database after bit of time.
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Jul 2026 06:11:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23125/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2026 19:47:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 3:
PS3 removes the pytest-xdist parts as with custom cluster tests it doesn't really work well anyway
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2026 19:52:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24544
to look at the new patch set (#4).
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Add a custom cluster test suite that exercises Impala <-> Trino
interoperability over Apache Iceberg V3 tables, plus the test-infra
support to drive and verify Trino from .test files.
Test infra:
- New TRINO_QUERY / RESULTS .test sections, analogous to
HIVE_QUERY / RESULTS. TRINO_QUERY statements run against the Trino
container (iceberg catalog, test database as schema); RESULTS
verifies the returned rows (with an optional VERIFY_* modifier).
- tests/common/trino_cluster.py: TrinoCluster lifecycle helper
(start/stop/is_container_running) that drives the Trino CLI inside the
'impala-minicluster-trino' container via 'docker exec', so no new
Python dependency is required. Results are formatted into the Impala
RESULTS textual convention and compared as opaque strings. start()
builds the image (build-trino-docker-image.sh) lazily if it is missing
and starts the container, so the Docker cost is paid only by runs that
actually execute the interop tests (mirrors the MySQL JDBC tests) and
never by data loading, cluster setup or run-all-tests.sh.
- run_test_case dispatches TRINO_QUERY to Trino and verifies against
RESULTS; run_stmt_in_trino mirrors run_stmt_in_hive.
- CustomClusterTestSuite.with_args(run_trino=True): the framework starts
the Trino container before the Impala cluster and stops it on teardown
(only if it started it), so interop suites no longer hand-roll
setup_class/teardown_class. If the Impala setup fails after Trino
started, Trino is stopped so a failed setup does not leak the
container.
- Disable Trino's DELTA_LENGTH_BYTE_ARRAY Parquet encoding for the
Iceberg catalog, which Impala cannot read for string/binary columns
yet (Trino 482 defaults it on). This is applied server-side via the
image's file session property manager (session-property-config.*),
setting parquet_writer_delta_length_byte_array_encoding_enabled=false.
- Set hive.non-managed-table-writes-enabled=true in the image's
hive.properties. Our HMS runs the default metadata transformer, which
converts the MANAGED tables Trino requests into EXTERNAL ones (Trino
does not advertise Hive ACID write capabilities), and Trino refuses to
write into existing external tables by default; enabling this lets the
hive-catalog interop test use plain CREATE TABLE + INSERT.
Tests (run in run-tests.py's serial phase like other custom cluster
tests, skipped only on non-HDFS filesystems, and failing rather than
skipping if the Trino container/Docker is unavailable so problems are
visible):
- tests/custom_cluster/test_iceberg_trino_interop.py (iceberg catalog):
- tests/custom_cluster/test_trino_interop.py (hive catalog): a minimal
suite over legacy (non-Iceberg) Hive tables
Pin the Trino Docker image to trinodb/trino:482 instead of 'latest' so
the interop tests are reproducible and not made flaky by future Trino
releases (482 is the first release with full Iceberg V3 support).
Docs: testdata/bin/TRINO-README.md explains how to run the suites and
author TRINO_QUERY / RESULTS sections.
Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Assisted-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
M testdata/bin/TRINO-README.md
M testdata/bin/minicluster_trino/Dockerfile
M testdata/bin/minicluster_trino/hive.properties
A testdata/bin/minicluster_trino/session-property-config.json
A testdata/bin/minicluster_trino/session-property-config.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-default-values.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-delete-dv.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-insert.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/test_result_verifier.py
A tests/common/trino_cluster.py
A tests/custom_cluster/test_iceberg_trino_interop.py
A tests/custom_cluster/test_trino_interop.py
M tests/util/test_file_parser.py
15 files changed, 892 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/24544/4
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23084/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 17:10:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 21:18:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 3:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py
File tests/custom_cluster/test_trino_interop.py:
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@59
PS3, Line 59: v
flake8: U100 Unused argument 'vector'
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@88
PS3, Line 88: v
flake8: U100 Unused argument 'vector'
http://gerrit.cloudera.org:8080/#/c/24544/3/tests/custom_cluster/test_trino_interop.py@110
PS3, Line 110: )
flake8: E501 line too long (92 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2026 19:52:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23129/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2026 20:12:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24544
to look at the new patch set (#3).
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Add a custom cluster test suite that exercises Impala <-> Trino
interoperability over Apache Iceberg V3 tables, plus the test-infra
support to drive and verify Trino from .test files.
Test infra:
- New TRINO_QUERY / RESULTS .test sections, analogous to
HIVE_QUERY / RESULTS. TRINO_QUERY statements run against the Trino
container (iceberg catalog, test database as schema); RESULTS
verifies the returned rows (with an optional VERIFY_* modifier).
- tests/common/trino_cluster.py: TrinoCluster lifecycle helper
(start/stop/is_container_running) that drives the Trino CLI inside the
'impala-minicluster-trino' container via 'docker exec', so no new
Python dependency is required. Results are formatted into the Impala
RESULTS textual convention and compared as opaque strings. start()
builds the image (build-trino-docker-image.sh) lazily if it is missing
and starts the container, so the Docker cost is paid only by runs that
actually execute the interop tests (mirrors the MySQL JDBC tests) and
never by data loading, cluster setup or run-all-tests.sh.
- run_test_case dispatches TRINO_QUERY to Trino and verifies against
RESULTS; run_stmt_in_trino mirrors run_stmt_in_hive.
- CustomClusterTestSuite.with_args(run_trino=True): the framework starts
the Trino container before the Impala cluster and stops it on teardown
(only if it started it), so interop suites no longer hand-roll
setup_class/teardown_class. If the Impala setup fails after Trino
started, Trino is stopped so a failed setup does not leak the
container.
- Disable Trino's DELTA_LENGTH_BYTE_ARRAY Parquet encoding for the
Iceberg catalog, which Impala cannot read for string/binary columns
yet (Trino 482 defaults it on). This is applied server-side via the
image's file session property manager (session-property-config.*),
setting parquet_writer_delta_length_byte_array_encoding_enabled=false.
- Set hive.non-managed-table-writes-enabled=true in the image's
hive.properties. Our HMS runs the default metadata transformer, which
converts the MANAGED tables Trino requests into EXTERNAL ones (Trino
does not advertise Hive ACID write capabilities), and Trino refuses to
write into existing external tables by default; enabling this lets the
hive-catalog interop test use plain CREATE TABLE + INSERT.
Tests (run in run-tests.py's serial phase like other custom cluster
tests, skipped only on non-HDFS filesystems, and failing rather than
skipping if the Trino container/Docker is unavailable so problems are
visible):
- tests/custom_cluster/test_iceberg_trino_interop.py (iceberg catalog):
- tests/custom_cluster/test_trino_interop.py (hive catalog): a minimal
suite over legacy (non-Iceberg) Hive tables
Pin the Trino Docker image to trinodb/trino:482 instead of 'latest' so
the interop tests are reproducible and not made flaky by future Trino
releases (482 is the first release with full Iceberg V3 support).
Docs: testdata/bin/TRINO-README.md explains how to run the suites and
author TRINO_QUERY / RESULTS sections.
Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Assisted-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
M testdata/bin/TRINO-README.md
M testdata/bin/minicluster_trino/Dockerfile
M testdata/bin/minicluster_trino/hive.properties
A testdata/bin/minicluster_trino/session-property-config.json
A testdata/bin/minicluster_trino/session-property-config.properties
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-default-values.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-delete-dv.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-insert.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/test_result_verifier.py
A tests/common/trino_cluster.py
A tests/custom_cluster/test_iceberg_trino_interop.py
A tests/custom_cluster/test_trino_interop.py
M tests/util/test_file_parser.py
15 files changed, 867 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/24544/3
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23136/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Jul 2026 11:52:11 +0000
Gerrit-HasComments: No