You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2022/07/20 05:52:58 UTC

[Impala-ASF-CR] IMPALA-11445: Allow partitions located in different file systems

Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18759


Change subject: IMPALA-11445: Allow partitions located in different file systems
......................................................................

IMPALA-11445: Allow partitions located in different file systems

When adding a partition with location in a file system which is
different from the file system of the table location, Impala accept
it. But when insert values to the table, catalogd throw exception.

This patch fix the issue by using the right FileSystem object.

Testing:
 - Added new test cases with partitions on different file systems.
 - Did manual tests in cluster with partitions on HDFS and Ozone.
 - Passed core test.

Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
---
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A testdata/workloads/functional-query/queries/QueryTest/partition-location.test
M tests/query_test/test_partitioning.py
4 files changed, 117 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/18759/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................

IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

When adding a partition with location in a file system which is
different from the file system of the table location, Impala accept
it. But when insert values to the table, catalogd throw exception.

This patch fix the issue by using the right FileSystem object.

Testing:
- Added new test case with partitions on different file systems.
Ran the test on S3.
- Did manual tests in cluster with partitions on HDFS and Ozone.
- Passed core test.

Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
---
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A testdata/workloads/functional-query/queries/QueryTest/partition-location.test
M tests/query_test/test_partitioning.py
4 files changed, 120 insertions(+), 3 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/18759/2
--
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 7: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8349/


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 19:27:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................

IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

When adding a partition with location in a file system which is
different from the file system of the table location, Impala accept
it. But when insert values to the table, catalogd throw exception.

This patch fix the issue by using the right FileSystem object.

Testing:
 - Added new test case with partitions on different file systems.
   Ran the test on S3.
 - Did manual tests in cluster with partitions on HDFS and Ozone.
 - Passed core test.

Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Reviewed-on: http://gerrit.cloudera.org:8080/18759
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A testdata/workloads/functional-query/queries/QueryTest/partition-location.test
A tests/custom_cluster/test_partition.py
4 files changed, 187 insertions(+), 2 deletions(-)

Approvals:
  Quanlong Huang: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18759/4/tests/custom_cluster/test_partition.py
File tests/custom_cluster/test_partition.py:

http://gerrit.cloudera.org:8080/#/c/18759/4/tests/custom_cluster/test_partition.py@25
PS4, Line 25: c
> flake8: E126 continuation line over-indented for hanging indent
Done


http://gerrit.cloudera.org:8080/#/c/18759/4/tests/custom_cluster/test_partition.py@50
PS4, Line 50: i
> flake8: E126 continuation line over-indented for hanging indent
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 08:46:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8349/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 14:39:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8346/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 09:15:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................

IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

When adding a partition with location in a file system which is
different from the file system of the table location, Impala accept
it. But when insert values to the table, catalogd throw exception.

This patch fix the issue by using the right FileSystem object.

Testing:
- Added new test case with partitions on different file systems.
Ran the test on S3.
- Did manual tests in cluster with partitions on HDFS and Ozone.
- Passed core test.

Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
---
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A testdata/workloads/functional-query/queries/QueryTest/partition-location.test
A tests/custom_cluster/test_partition.py
4 files changed, 187 insertions(+), 2 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/18759/8
--
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 8: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 23:25:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18759/4/tests/custom_cluster/test_partition.py
File tests/custom_cluster/test_partition.py:

http://gerrit.cloudera.org:8080/#/c/18759/4/tests/custom_cluster/test_partition.py@25
PS4, Line 25: c
flake8: E126 continuation line over-indented for hanging indent


http://gerrit.cloudera.org:8080/#/c/18759/4/tests/custom_cluster/test_partition.py@50
PS4, Line 50: i
flake8: E126 continuation line over-indented for hanging indent



-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 08:37:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18759/5/tests/custom_cluster/test_partition.py
File tests/custom_cluster/test_partition.py:

http://gerrit.cloudera.org:8080/#/c/18759/5/tests/custom_cluster/test_partition.py@44
PS5, Line 44: c
flake8: E126 continuation line over-indented for hanging indent



-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 08:48:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8344/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jul 2022 22:44:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11003/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 09:07:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................

IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

When adding a partition with location in a file system which is
different from the file system of the table location, Impala accept
it. But when insert values to the table, catalogd throw exception.

This patch fix the issue by using the right FileSystem object.

Testing:
- Added new test case with partitions on different file systems.
Ran the test on S3.
- Did manual tests in cluster with partitions on HDFS and Ozone.
- Passed core test.

Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
---
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A testdata/workloads/functional-query/queries/QueryTest/partition-location.test
A tests/custom_cluster/test_partition.py
4 files changed, 169 insertions(+), 2 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/18759/6
--
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................

IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

When adding a partition with location in a file system which is
different from the file system of the table location, Impala accept
it. But when insert values to the table, catalogd throw exception.

This patch fix the issue by using the right FileSystem object.

Testing:
- Added new test case with partitions on different file systems.
Ran the test on S3.
- Did manual tests in cluster with partitions on HDFS and Ozone.
- Passed core test.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/18759/3
--
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-11445: Allow partitions located in different file systems

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Allow partitions located in different file systems
......................................................................


Patch Set 1: Code-Review+1

(3 comments)

Thanks for fixing this quickly! The patch LGTM. I just have some minor comments.

http://gerrit.cloudera.org:8080/#/c/18759/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18759/1//COMMIT_MSG@7
PS1, Line 7: Allow partitions located in different file systems
I think we already allow this but we have bug in firing insert events. Can we rework this to something like "Fix bug in firing insert events of partitions located in different FS"?


http://gerrit.cloudera.org:8080/#/c/18759/1/testdata/workloads/functional-query/queries/QueryTest/partition-location.test
File testdata/workloads/functional-query/queries/QueryTest/partition-location.test:

http://gerrit.cloudera.org:8080/#/c/18759/1/testdata/workloads/functional-query/queries/QueryTest/partition-location.test@72
PS1, Line 72: hdfs://localhost:20500
Can we run the test on S3 if we replace this to $FILESYSTEM_PREFIX?


http://gerrit.cloudera.org:8080/#/c/18759/1/tests/query_test/test_partitioning.py
File tests/query_test/test_partitioning.py:

http://gerrit.cloudera.org:8080/#/c/18759/1/tests/query_test/test_partitioning.py@100
PS1, Line 100:   @SkipIfS3.hive
             :   @SkipIfGCS.hive
             :   @SkipIfCOS.hive
             :   @SkipIfABFS.hive
             :   @SkipIfADLS.hive
It'd be nice if we can test this on object storages. Let's see if using $FILESYSTEM_PREFIX in the test file works.



-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 20 Jul 2022 14:00:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 8: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Jul 2022 04:23:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 7:

The new test use local file system for partition locations. But verification job create impala mini-cluster with calatogd, statestored and impalad running on different containers. This cause calatogd could not access the data files generated by impalad.
Switch to local-catalog mode for the new test.


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 22:42:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Allow partitions located in different file systems

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Allow partitions located in different file systems
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10993/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 20 Jul 2022 06:13:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18759/2/tests/query_test/test_partitioning.py
File tests/query_test/test_partitioning.py:

http://gerrit.cloudera.org:8080/#/c/18759/2/tests/query_test/test_partitioning.py@111
PS2, Line 111: 
flake8: W292 no newline at end of file



-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jul 2022 22:23:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 14:00:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Allow partitions located in different file systems

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Allow partitions located in different file systems
......................................................................


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/18759/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18759/1//COMMIT_MSG@7
PS1, Line 7: Allow partitions located in different file systems
> I think we already allow this but we have bug in firing insert events. Can 
Done


http://gerrit.cloudera.org:8080/#/c/18759/1/testdata/workloads/functional-query/queries/QueryTest/partition-location.test
File testdata/workloads/functional-query/queries/QueryTest/partition-location.test:

http://gerrit.cloudera.org:8080/#/c/18759/1/testdata/workloads/functional-query/queries/QueryTest/partition-location.test@72
PS1, Line 72: hdfs://localhost:20500
> Can we run the test on S3 if we replace this to $FILESYSTEM_PREFIX?
Changed to $FILESYSTEM_PREFIX and ran the test on s3


http://gerrit.cloudera.org:8080/#/c/18759/1/tests/query_test/test_partitioning.py
File tests/query_test/test_partitioning.py:

http://gerrit.cloudera.org:8080/#/c/18759/1/tests/query_test/test_partitioning.py@100
PS1, Line 100:   @SkipIfS3.hive
             :   @SkipIfGCS.hive
             :   @SkipIfCOS.hive
             :   @SkipIfABFS.hive
             :   @SkipIfADLS.hive
> It'd be nice if we can test this on object storages. Let's see if using $FI
Changed to $FILESYSTEM_PREFIX and could run the test on s3.



-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jul 2022 22:20:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................

IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

When adding a partition with location in a file system which is
different from the file system of the table location, Impala accept
it. But when insert values to the table, catalogd throw exception.

This patch fix the issue by using the right FileSystem object.

Testing:
- Added new test case with partitions on different file systems.
Ran the test on S3.
- Did manual tests in cluster with partitions on HDFS and Ozone.
- Passed core test.

Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
---
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A testdata/workloads/functional-query/queries/QueryTest/partition-location.test
A tests/custom_cluster/test_partition.py
4 files changed, 169 insertions(+), 2 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/18759/5
--
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8352/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 23:30:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8344/


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 03:31:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11002/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 08:56:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 3: Code-Review+2

LGTM. Thanks for addressing the comments!


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jul 2022 23:44:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11004/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 09:12:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11000/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jul 2022 22:42:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11013/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 23:44:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 7:

(1 comment)

> Patch Set 7:
> 
> The new test use local file system for partition locations. But verification job create impala mini-cluster with calatogd, statestored and impalad running on different containers. This cause calatogd could not access the data files generated by impalad.
> Switch to local-catalog mode for the new test.

Nice finding! Let's move this to the custom cluster test and add the coverage on both catalog modes.

http://gerrit.cloudera.org:8080/#/c/18759/6/tests/custom_cluster/test_partition.py
File tests/custom_cluster/test_partition.py:

http://gerrit.cloudera.org:8080/#/c/18759/6/tests/custom_cluster/test_partition.py@52
PS6, Line 52: 
We also need to test this on legacy catalog mode, i.e. 

  @CustomClusterTestSuite.with_args(
    impalad_args="--use_local_catalog=false",
    catalogd_args="--catalog_topic_mode=full")



-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 22:54:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11001/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jul 2022 22:49:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11009/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 14:59:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................

IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

When adding a partition with location in a file system which is
different from the file system of the table location, Impala accept
it. But when insert values to the table, catalogd throw exception.

This patch fix the issue by using the right FileSystem object.

Testing:
- Added new test case with partitions on different file systems.
Ran the test on S3.
- Did manual tests in cluster with partitions on HDFS and Ozone.
- Passed core test.

Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
---
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A testdata/workloads/functional-query/queries/QueryTest/partition-location.test
A tests/custom_cluster/test_partition.py
4 files changed, 169 insertions(+), 2 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/18759/4
--
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18759/5/tests/custom_cluster/test_partition.py
File tests/custom_cluster/test_partition.py:

http://gerrit.cloudera.org:8080/#/c/18759/5/tests/custom_cluster/test_partition.py@44
PS5, Line 44: c
> flake8: E126 continuation line over-indented for hanging indent
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Jul 2022 08:56:25 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18759 )

Change subject: IMPALA-11445: Fix bug in firing insert event of partitions located in different FS
......................................................................

IMPALA-11445: Fix bug in firing insert event of partitions located in different FS

When adding a partition with location in a file system which is
different from the file system of the table location, Impala accept
it. But when insert values to the table, catalogd throw exception.

This patch fix the issue by using the right FileSystem object.

Testing:
- Added new test case with partitions on different file systems.
Ran the test on S3.
- Did manual tests in cluster with partitions on HDFS and Ozone.
- Passed core test.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/18759/7
--
To view, visit http://gerrit.cloudera.org:8080/18759
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0491ee1bf40c3d5240f9124cef3f3169c44a8267
Gerrit-Change-Number: 18759
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>