You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2022/11/28 18:32:00 UTC

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19262


Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................

IMPALA-11736: Copy data between ofs buckets

When Impala moves a file - such as for a LOAD DATA statement - it checks
whether the source and destination are the same filesystem. If the same,
it uses hdfsRename, otherwise it uses hdfsMove to move between
filesystems.

Ozone's ofs protocol supports referencing multiple buckets by path in
the same filesystem, but does not support rename between them. All other
filesystems Impala supports include the bucket name (if they use that
concept) in the authority. This patch updates the function used to
determine whether two paths are in the same filesystem to also check
that they're in the same bucket as a requirement for hdfsRename.

Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
---
M be/src/runtime/dml-exec-state.cc
M be/src/runtime/exec-env.h
M be/src/util/hdfs-util-test.cc
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
M tests/metadata/test_load.py
8 files changed, 198 insertions(+), 59 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/19262/2
-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19262

to look at the new patch set (#6).

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................

IMPALA-11736: Copy data between ofs buckets

When Impala moves a file - such as for a LOAD DATA statement - it checks
whether the source and destination are the same filesystem. If the same,
it uses hdfsRename, otherwise it uses hdfsMove to move between
filesystems.

Ozone's ofs protocol supports referencing multiple buckets by path in
the same filesystem, but does not support rename between them. All other
filesystems Impala supports include the bucket name (if they use that
concept) in the authority. This patch updates the function used to
determine whether two paths are in the same filesystem to also check
that they're in the same bucket as a requirement for hdfsRename.

Testing: ran test suite with Ozone.

Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
---
M be/src/runtime/dml-exec-state.cc
M be/src/runtime/exec-env.h
M be/src/util/hdfs-util-test.cc
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
M tests/metadata/test_load.py
8 files changed, 199 insertions(+), 59 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/19262/6
-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................

IMPALA-11736: Copy data between ofs buckets

When Impala moves a file - such as for a LOAD DATA statement - it checks
whether the source and destination are the same filesystem. If the same,
it uses hdfsRename, otherwise it uses hdfsMove to move between
filesystems.

Ozone's ofs protocol supports referencing multiple buckets by path in
the same filesystem, but does not support rename between them. All other
filesystems Impala supports include the bucket name (if they use that
concept) in the authority. This patch updates the function used to
determine whether two paths are in the same filesystem to also check
that they're in the same bucket as a requirement for hdfsRename.

Testing: ran test suite with Ozone.

Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Reviewed-on: http://gerrit.cloudera.org:8080/19262
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/runtime/dml-exec-state.cc
M be/src/runtime/exec-env.h
M be/src/util/hdfs-util-test.cc
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
M tests/metadata/test_load.py
8 files changed, 199 insertions(+), 59 deletions(-)

Approvals:
  Joe McDonnell: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8845/


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Dec 2022 00:39:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11982/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Dec 2022 22:24:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11939/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Dec 2022 19:44:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11914/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 19:02:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8874/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Dec 2022 01:38:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:

http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@310
PS5, Line 310:     boolean doRename = destIsDfs && sameEncryptionZone;
> I think we want to keep an equivalent to the "sameFileSystem" condition her
sameEncryptionZone makes that redundant, but I guess it wouldn't hurt to keep around.


http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@315
PS5, Line 315:       LOG.trace("Moving '{}' to '{}'", sourceFile.toString(), destFile.toString());
> Just to check my understanding: Are the "LOG.isTraceEnabled()" statements u
I need to remove toString, that's implied by the conversion and in fact skipped later in this same function.

My understanding of the conditional pattern is that it's to avoid string formatting when you're not going to use the results. However https://www.slf4j.org/faq.html#logging_performance discusses that; if you use slf4j's string substitution and avoid method calls, there's no overhead to calling LOG.trace directly vs guarding it with a conditional if you're not logging at that level.


http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
File fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java:

http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java@20
PS5, Line 20: import org.apache.commons.lang3.tuple.Pair;
> Nit: Could you switch this to use org.apache.impala.common.Pair? (Just to a
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Dec 2022 21:42:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Dec 2022 06:48:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19262

to look at the new patch set (#3).

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................

IMPALA-11736: Copy data between ofs buckets

When Impala moves a file - such as for a LOAD DATA statement - it checks
whether the source and destination are the same filesystem. If the same,
it uses hdfsRename, otherwise it uses hdfsMove to move between
filesystems.

Ozone's ofs protocol supports referencing multiple buckets by path in
the same filesystem, but does not support rename between them. All other
filesystems Impala supports include the bucket name (if they use that
concept) in the authority. This patch updates the function used to
determine whether two paths are in the same filesystem to also check
that they're in the same bucket as a requirement for hdfsRename.

Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
---
M be/src/runtime/dml-exec-state.cc
M be/src/runtime/exec-env.h
M be/src/util/hdfs-util-test.cc
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
M tests/metadata/test_load.py
8 files changed, 199 insertions(+), 59 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/19262/3
-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/19262/2/be/src/util/hdfs-util-test.cc
File be/src/util/hdfs-util-test.cc:

http://gerrit.cloudera.org:8080/#/c/19262/2/be/src/util/hdfs-util-test.cc@56
PS2, Line 56:   EXPECT_FALSE(FilesystemsAndBucketsMatch("hdfs://", "s3a://dummybucket/temp_dir/temp_path"));
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/19262/2/tests/metadata/test_load.py
File tests/metadata/test_load.py:

http://gerrit.cloudera.org:8080/#/c/19262/2/tests/metadata/test_load.py@41
PS2, Line 41: =
flake8: E225 missing whitespace around operator



-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 18:32:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8845/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Nov 2022 19:32:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11913/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 18:51:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Dec 2022 00:30:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19262

to look at the new patch set (#5).

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................

IMPALA-11736: Copy data between ofs buckets

When Impala moves a file - such as for a LOAD DATA statement - it checks
whether the source and destination are the same filesystem. If the same,
it uses hdfsRename, otherwise it uses hdfsMove to move between
filesystems.

Ozone's ofs protocol supports referencing multiple buckets by path in
the same filesystem, but does not support rename between them. All other
filesystems Impala supports include the bucket name (if they use that
concept) in the authority. This patch updates the function used to
determine whether two paths are in the same filesystem to also check
that they're in the same bucket as a requirement for hdfsRename.

Testing: ran test suite with Ozone.

Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
---
M be/src/runtime/dml-exec-state.cc
M be/src/runtime/exec-env.h
M be/src/util/hdfs-util-test.cc
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
M tests/metadata/test_load.py
8 files changed, 199 insertions(+), 59 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/19262/5
-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8850/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Dec 2022 19:25:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 6: Code-Review+2

(1 comment)

This looks good to me

http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:

http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@310
PS5, Line 310:     boolean doRename = destIsDfs && sameBucket && sameE
> sameEncryptionZone makes that redundant, but I guess it wouldn't hurt to ke
Yeah, I think the logic in arePathsInSameHdfsEncryptionZone() is pretty minimal if encryption is off.



-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Dec 2022 00:50:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11736: Copy data between ofs buckets

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19262 )

Change subject: IMPALA-11736: Copy data between ofs buckets
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:

http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@310
PS5, Line 310:     boolean doRename = destIsDfs && sameEncryptionZone;
I think we want to keep an equivalent to the "sameFileSystem" condition here (sameBucket would be fine).


http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@315
PS5, Line 315:       LOG.trace("Moving '{}' to '{}'", sourceFile.toString(), destFile.toString());
Just to check my understanding: Are the "LOG.isTraceEnabled()" statements unnecessary because they are already controlled by the log level?


http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
File fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java:

http://gerrit.cloudera.org:8080/#/c/19262/5/fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java@20
PS5, Line 20: import org.apache.commons.lang3.tuple.Pair;
Nit: Could you switch this to use org.apache.impala.common.Pair? (Just to avoid an extra external dependency)



-- 
To view, visit http://gerrit.cloudera.org:8080/19262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic61f01672fa605fec0377885b13a1621573e424e
Gerrit-Change-Number: 19262
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Dec 2022 18:59:32 +0000
Gerrit-HasComments: Yes