You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org> on 2023/01/11 15:13:44 UTC
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length in case of metadata rewrite
Gergely Fürnstáhl has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19412
Change subject: IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file pathes. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Testing:
- Tested locally, manually
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
---
M testdata/bin/rewrite-iceberg-metadata.py
1 file changed, 24 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/19412/1
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 4:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/19412/4/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/4/testdata/bin/rewrite-iceberg-metadata.py@38
PS4, Line 38: def generate_new_path(prefix, file_path):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/19412/4/testdata/bin/rewrite-iceberg-metadata.py@44
PS4, Line 44: )
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/19412/4/testdata/bin/rewrite-iceberg-metadata.py@52
PS4, Line 52: s
flake8: E501 line too long (94 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/19412/4/testdata/bin/rewrite-iceberg-metadata.py@65
PS4, Line 65: a
flake8: E501 line too long (96 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/19412/4/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/19412/4/tests/query_test/test_iceberg.py@788
PS4, Line 788:
flake8: W293 blank line contains whitespace
http://gerrit.cloudera.org:8080/#/c/19412/4/tests/query_test/test_iceberg.py@788
PS4, Line 788:
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 4
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 09:30:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 12: Code-Review+1
Nice patch, thank you!
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 12
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Jan 2023 13:36:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file pathes. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Additionally, improved path rewrite to be able to rewrite absolute
paths correctly.
Testing:
- Tested locally, manually
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
---
M testdata/bin/rewrite-iceberg-metadata.py
M tests/common/skip.py
M tests/query_test/test_iceberg.py
3 files changed, 41 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/19412/4
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 4
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19412/7/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/7/testdata/bin/rewrite-iceberg-metadata.py@35
PS7, Line 35: pathes
> nit: paths
Done
http://gerrit.cloudera.org:8080/#/c/19412/7/testdata/bin/rewrite-iceberg-metadata.py@40
PS7, Line 40: pathes
> nit: paths
Done
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 7
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 13:19:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12172/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 9
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 13:39:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12168/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 6
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 10:06:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12173/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 10
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 13:39:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8969/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 12
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Jan 2023 17:19:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 5:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/19412/5/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/5/testdata/bin/rewrite-iceberg-metadata.py@38
PS5, Line 38: def generate_new_path(prefix, file_path):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/19412/5/testdata/bin/rewrite-iceberg-metadata.py@44
PS5, Line 44: )
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/19412/5/testdata/bin/rewrite-iceberg-metadata.py@52
PS5, Line 52: s
flake8: E501 line too long (94 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/19412/5/testdata/bin/rewrite-iceberg-metadata.py@65
PS5, Line 65: a
flake8: E501 line too long (96 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/19412/5/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/19412/5/tests/query_test/test_iceberg.py@788
PS5, Line 788:
flake8: W293 blank line contains whitespace
http://gerrit.cloudera.org:8080/#/c/19412/5/tests/query_test/test_iceberg.py@788
PS5, Line 788:
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 5
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 09:41:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19412/1/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/1/testdata/bin/rewrite-iceberg-metadata.py@121
PS1, Line 121:
flake8: W292 no newline at end of file
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Jan 2023 15:14:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length in case of metadata rewrite
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
......................................................................
Patch Set 1: Code-Review+1
IMPALA-11821 talks about reverting https://gerrit.cloudera.org/c/19387/ as well. When will that be done?
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Jan 2023 17:00:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 12: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 12
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Jan 2023 17:18:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file paths. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Additionally, improved path rewrite to be able to rewrite absolute
paths correctly and pretty dumping metadata jsons.
Testing:
- Tested locally, manually verified the rewrites
- Tested on Ozone, automatically rewriting the test data and running
test_iceberg.py
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
---
M testdata/bin/rewrite-iceberg-metadata.py
M tests/common/skip.py
M tests/query_test/test_iceberg.py
3 files changed, 47 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/19412/12
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 12
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file pathes. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Additionally, improved path rewrite to be able to rewrite absolute
paths correctly and pretty dumping metadata jsons.
Testing:
- Tested locally, manually
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
---
M testdata/bin/rewrite-iceberg-metadata.py
M tests/common/skip.py
M tests/query_test/test_iceberg.py
3 files changed, 47 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/19412/9
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 9
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file pathes. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Additionally, improved path rewrite to be able to rewrite absolute
paths correctly.
Testing:
- Tested locally, manually
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
---
M testdata/bin/rewrite-iceberg-metadata.py
M tests/common/skip.py
M tests/query_test/test_iceberg.py
3 files changed, 41 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/19412/5
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 5
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12169/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 7
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 11:14:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file pathes. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Testing:
- Tested locally, manually
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
---
M testdata/bin/rewrite-iceberg-metadata.py
1 file changed, 24 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/19412/2
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12167/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 5
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 10:00:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file pathes. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Additionally, improved path rewrite to be able to rewrite absolute
paths correctly.
Testing:
- Tested locally, manually
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
---
M testdata/bin/rewrite-iceberg-metadata.py
M tests/common/skip.py
M tests/query_test/test_iceberg.py
3 files changed, 45 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/19412/7
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 7
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file paths. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Additionally, improved path rewrite to be able to rewrite absolute
paths correctly and pretty dumping metadata jsons.
Testing:
- Tested locally, manually verified the rewrites
- Tested on Ozone, automatically rewriting the test data and running
test_iceberg.py
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Reviewed-on: http://gerrit.cloudera.org:8080/19412
Reviewed-by: Noemi Pap-Takacs <np...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M testdata/bin/rewrite-iceberg-metadata.py
M tests/common/skip.py
M tests/query_test/test_iceberg.py
3 files changed, 47 insertions(+), 11 deletions(-)
Approvals:
Noemi Pap-Takacs: Looks good to me, but someone else must approve
Michael Smith: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 13
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19412/1/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/1/testdata/bin/rewrite-iceberg-metadata.py@121
PS1, Line 121:
> Should address this.
Done
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 09:29:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 7:
(3 comments)
Thank you for reverting the 'SkipIf' annotations in the avro and mixed format tests, and also for the path rewrites.
'Location' also needs to be rewritten.
http://gerrit.cloudera.org:8080/#/c/19412/7/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/7/testdata/bin/rewrite-iceberg-metadata.py@35
PS7, Line 35: pathes
nit: paths
http://gerrit.cloudera.org:8080/#/c/19412/7/testdata/bin/rewrite-iceberg-metadata.py@40
PS7, Line 40: pathes
nit: paths
http://gerrit.cloudera.org:8080/#/c/19412/7/testdata/bin/rewrite-iceberg-metadata.py@97
PS7, Line 97: prefix + metadata['location']
generate_new_path(prefix, metadata['location'])
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 7
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 12:52:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12158/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Jan 2023 17:59:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file pathes. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Additionally, improved path rewrite to be able to rewrite absolute
paths correctly and pretty dumping metadata jsons.
Testing:
- Tested locally, manually
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
---
M testdata/bin/rewrite-iceberg-metadata.py
M tests/common/skip.py
M tests/query_test/test_iceberg.py
3 files changed, 47 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/19412/10
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 10
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19412/7/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/7/testdata/bin/rewrite-iceberg-metadata.py@97
PS7, Line 97: prefix + metadata['location']
> generate_new_path(prefix, metadata['location'])
Thanks, nice catch! Fixed
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 7
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 13:18:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
testdata/bin/rewrite-iceberg-metadata.py rewrites manifest and snapshot
files using the provided prefix for file pathes. Snapshot files store
the length of manifest files as well, this needs to be adjusted too.
Additionally, improved path rewrite to be able to rewrite absolute
paths correctly.
Testing:
- Tested locally, manually
Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
---
M testdata/bin/rewrite-iceberg-metadata.py
M tests/common/skip.py
M tests/query_test/test_iceberg.py
3 files changed, 44 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/19412/6
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 6
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length in case of metadata rewrite
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
......................................................................
Patch Set 1: Code-Review+2
This seems fine.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Jan 2023 16:56:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length in case of metadata rewrite
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
......................................................................
Patch Set 2:
(2 comments)
Looks nice. I added some suggestions so that this script could handle tables created by Hive as well.
That would solve IMPALA-11807.
http://gerrit.cloudera.org:8080/#/c/19412/2/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/2/testdata/bin/rewrite-iceberg-metadata.py@69
PS2, Line 69: for mfile in glob.glob(os.path.join(arg, 'v*.metadata.json')):
I think we should remove 'v' from the regex to also match metadata.json files created by Hive.
http://gerrit.cloudera.org:8080/#/c/19412/2/testdata/bin/rewrite-iceberg-metadata.py@83
PS2, Line 83: metadata['location'] = prefix + metadata['location']
Could you please add support for absolute paths?
Iceberg creates metadata files with absolute paths. We used to rewrite metadata files manually to have relative paths instead. This script currently only adds filesystem prefixes to relative paths. It would be nice if it could handle changing absolute paths as well: removing the old prefix (everything before '/test-warehouse/iceberg_test') and adding the new one. Then we would not need to manually rewrite metadata and we could create test tables using Hive from Impala directly.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 2
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Sun, 15 Jan 2023 20:32:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12177/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 12
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Jan 2023 13:56:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length in case of metadata rewrite
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19412/1/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/1/testdata/bin/rewrite-iceberg-metadata.py@121
PS1, Line 121:
> flake8: W292 no newline at end of file
Should address this.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Jan 2023 17:00:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 12: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 12
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Jan 2023 22:33:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length in case of metadata rewrite
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12148/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Jan 2023 15:33:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12166/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 4
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 09:50:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11821: Adjusting manifest length and absolute paths in case of metadata rewrite
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19412 )
Change subject: IMPALA-11821: Adjusting manifest_length and absolute paths in case of metadata rewrite
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19412/2/testdata/bin/rewrite-iceberg-metadata.py
File testdata/bin/rewrite-iceberg-metadata.py:
http://gerrit.cloudera.org:8080/#/c/19412/2/testdata/bin/rewrite-iceberg-metadata.py@69
PS2, Line 69: def fix_manifest_length(entry):
> I think we should remove 'v' from the regex to also match metadata.json fil
Done
http://gerrit.cloudera.org:8080/#/c/19412/2/testdata/bin/rewrite-iceberg-metadata.py@83
PS2, Line 83: if 'format-version' not in metadata:
> Could you please add support for absolute paths?
Done
--
To view, visit http://gerrit.cloudera.org:8080/19412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89b9208f25552012cc1ab16fa60a819dd5a683d9
Gerrit-Change-Number: 19412
Gerrit-PatchSet: 4
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Mon, 16 Jan 2023 09:30:22 +0000
Gerrit-HasComments: Yes