You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org> on 2023/01/19 16:19:35 UTC
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Gergely Fürnstáhl has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19432
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Iceberg test tables are usually written on hdfs and the file paths start
with "hdfs://localhost:20500/test-warehouse".
Earlier we manually transformed the metadata so paths would start with
"/test-warehouse"
Since IMPALA-11821, testdata/bin/rewrite-iceberg-metadata.py supports
not only a custom WAREHOUSE_LOCATION_PREFIX, but the ability to trim the
beginning of the file paths.
This commit modifies the data load, so metadata rewrite always executes
if not on hdfs, even with empty WAREHOUSE_LOCATION_PREFIX.
Testing:
- Ran iceberg tests on ozone and S3
Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
---
M testdata/bin/load-test-warehouse-snapshot.sh
1 file changed, 4 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/19432/1
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Jan 2023 18:36:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8980/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Jan 2023 08:06:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 1: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19432/1/testdata/bin/load-test-warehouse-snapshot.sh
File testdata/bin/load-test-warehouse-snapshot.sh:
http://gerrit.cloudera.org:8080/#/c/19432/1/testdata/bin/load-test-warehouse-snapshot.sh@120
PS1, Line 120: ${IMPALA_HOME}/testdata/bin/rewrite-iceberg-metadata.py "${WAREHOUSE_LOCATION_PREFIX}" \
So this removes the authority portion so we just have a path, even if WAREHOUSE_LOCATION_PREFIX is empty? That lets it work in S3. Makes sense to me.
http://gerrit.cloudera.org:8080/#/c/19432/1/testdata/bin/load-test-warehouse-snapshot.sh@121
PS1, Line 121: $(find ${SNAPSHOT_STAGING_DIR}${TEST_WAREHOUSE_DIR}/iceberg_test -name "metadata")
For future updates, I think it would be safe to search all files instead of just in iceberg_test.
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jan 2023 16:59:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 1: Code-Review+1
Thanks for the fix!
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jan 2023 17:25:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Iceberg test tables are usually written on hdfs and the file paths start
with "hdfs://localhost:20500/test-warehouse".
Earlier we manually transformed the metadata so paths would start with
"/test-warehouse"
Since IMPALA-11821, testdata/bin/rewrite-iceberg-metadata.py supports
not only a custom WAREHOUSE_LOCATION_PREFIX, but the ability to trim the
beginning of the file paths.
This commit modifies the data load, so metadata rewrite always executes
if not on hdfs, even with empty WAREHOUSE_LOCATION_PREFIX.
Testing:
- Ran iceberg tests on ozone and S3
Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Reviewed-on: http://gerrit.cloudera.org:8080/19432
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M testdata/bin/load-test-warehouse-snapshot.sh
1 file changed, 5 insertions(+), 3 deletions(-)
Approvals:
Michael Smith: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 3
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19432/1/testdata/bin/load-test-warehouse-snapshot.sh
File testdata/bin/load-test-warehouse-snapshot.sh:
http://gerrit.cloudera.org:8080/#/c/19432/1/testdata/bin/load-test-warehouse-snapshot.sh@120
PS1, Line 120: ${IMPALA_HOME}/testdata/bin/rewrite-iceberg-metadata.py "${WAREHOUSE_LOCATION_PREFIX}" \
> So this removes the authority portion so we just have a path, even if WAREH
Yes, exactly
http://gerrit.cloudera.org:8080/#/c/19432/1/testdata/bin/load-test-warehouse-snapshot.sh@121
PS1, Line 121: $(find ${SNAPSHOT_STAGING_DIR}${TEST_WAREHOUSE_DIR}/iceberg_test -name "metadata")
> For future updates, I think it would be safe to search all files instead of
Done
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 Jan 2023 17:21:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12204/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jan 2023 16:39:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8981/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Jan 2023 13:29:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8980/
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Jan 2023 13:13:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Iceberg test tables are usually written on hdfs and the file paths start
with "hdfs://localhost:20500/test-warehouse".
Earlier we manually transformed the metadata so paths would start with
"/test-warehouse"
Since IMPALA-11821, testdata/bin/rewrite-iceberg-metadata.py supports
not only a custom WAREHOUSE_LOCATION_PREFIX, but the ability to trim the
beginning of the file paths.
This commit modifies the data load, so metadata rewrite always executes
if not on hdfs, even with empty WAREHOUSE_LOCATION_PREFIX.
Testing:
- Ran iceberg tests on ozone and S3
Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
---
M testdata/bin/load-test-warehouse-snapshot.sh
1 file changed, 5 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/19432/2
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12214/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 Jan 2023 17:42:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11807: Rewrite iceberg metadata if not on hdfs
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19432 )
Change subject: IMPALA-11807: Rewrite iceberg metadata if not on hdfs
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/19432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic04c5abdd42cb0c1cf5abd310b06c39cf8cd64ba
Gerrit-Change-Number: 19432
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 Jan 2023 17:48:37 +0000
Gerrit-HasComments: No