You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org> on 2022/10/25 02:42:21 UTC
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19165
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/planner/ExplainTest.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
15 files changed, 90 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/1
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 1
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11696/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 2
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 03:14:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11814/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 13
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Tue, 08 Nov 2022 09:35:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has uploaded a new patch set (#18) to the change originally created by hezhenhua.hzh. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
This patch adds support for OSS (Aliyun Object Storage Service).
Using the hadoop-aliyun, the implementation is similar to other
remote FileSystems.
Tests:
- Prepare:
Initialize OSS-related environment variables:
OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT.
Compile and create hdfs test data on a ECS instance. Upload test data
to an OSS bucket.
- Modify all locations in HMS DB to point to the OSS bucket.
Remove some hdfs caching params. Run CORE tests.
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 87 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/18
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 18
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 18:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8810/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 18
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 16 Nov 2022 05:04:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#2).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/planner/ExplainTest.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
15 files changed, 90 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/2
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 2
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 17:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11856/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 17
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 16 Nov 2022 04:00:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11813/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 12
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Tue, 08 Nov 2022 09:00:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11698/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 5
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 03:44:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#10).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
This patch adds support for OSS (Aliyun Oobject Strorage Service).
Using the hadoop-aliyun, the implementation is similar to other remote FileSystems.
For more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service
Tests:
Run CORE tests.
Mdddanual ACID test verification.
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 91 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/10
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 10
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 18: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 18
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 16 Nov 2022 10:14:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#5).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/planner/ExplainTest.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
15 files changed, 94 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/5
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 5
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG@7
PS5, Line 7: Support Aliyun OSS File System
> Would be good to have a detailed commit message, including the description
thanks for your comment, i just run the core tests by, and do manual ACID test verification, not sure is it enough for you. btw, the cdp_components-31397203 missing hadoop-aliyun-*.jar and aliyun-sdk-oss-*.jar, should i raise a issue for this
?
http://gerrit.cloudera.org:8080/#/c/19165/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS5, Line 872: // Atomic operations: delete() and rename() are implemented by recursive
: // file-by-file operations.
> It is good to have such explanations for the atomicity of OSS. But not sure
you are right, rename a directory or a file in OSS are both not atomic. rename directory in OSS or in S3-like storage will be recursive file-by-file operations and rename a single file will be split into copy + delete. i will update this explanations.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 7
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 26 Oct 2022 08:42:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 7:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG@7
PS5, Line 7: Support Aliyun OSS File System
> thanks for your comment, i just run the core tests by, and do manual ACID
Done
http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG@9
PS7, Line 9: Add Aliyun OSS support
> This sentence is probably good to delete, otherwise a full stop is needed.
Done
http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG@11
PS7, Line 11: for
> nit: need a capital letter.
Done
http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG@14
PS7, Line 14: Manual ACID test verification
> nit: would be good to have a full stop.
Done
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 7
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Tue, 01 Nov 2022 06:28:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 13: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 13
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 09 Nov 2022 00:21:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11695/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 1
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 03:02:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11709/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 6
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 26 Oct 2022 09:02:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 16:
Hi Quanlong, PTAL when you are not busy.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 16
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Fri, 11 Nov 2022 02:39:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 13:
LGTM!
One minor thing is that, the commit message should be wrapped at 72 or fewer characters. See https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 13
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 09 Nov 2022 00:20:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11758/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 10
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Tue, 01 Nov 2022 06:17:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 11:
(5 comments)
The patch looks good. Aside from some typos, just have one more question about the testing in the commit message.
http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG@9
PS11, Line 9: Strorage
nit. Storage
http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG@9
PS11, Line 9: Oobject
nit. Object
http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG@14
PS11, Line 14: Mdddanual
nit. Manual
http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG@14
PS11, Line 14: Mdddanual ACID test verification.
Could you elaborate a bit on how did you test on this feature? For example, in IMPALA-7712: Support Google Cloud Storage, https://gerrit.cloudera.org/#/c/17121/, Quanlong described in the commit message that how he prepared for the GCS test data before running core tests. I think this description is useful when the test needs some manual work.
http://gerrit.cloudera.org:8080/#/c/19165/11/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:
http://gerrit.cloudera.org:8080/#/c/19165/11/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@447
PS11, Line 447: a
nit. an
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 11
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Tue, 01 Nov 2022 20:59:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 5:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG@7
PS5, Line 7: Support Aliyun OSS File System
> thanks for your comment, i just run the core tests by, and do manual ACID
I think they are not in the cdp_components-31397203. In my environment, I can find them in ~/.m2/repository. Probably a core pre review test with all tests passed would be good.
http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG@9
PS7, Line 9: Change-Id: I267e6531da5
This sentence is probably good to delete, otherwise a full stop is needed.
http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG@11
PS7, Line 11:
nit: need a capital letter.
http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG@14
PS7, Line 14:
nit: would be good to have a full stop.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 5
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 26 Oct 2022 17:17:13 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 2:
(14 comments)
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS2, Line 872: // Atomic operations: delete() and rename() are implemented by recursive
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@873
PS2, Line 873: // file-by-file operations. They take time at least proportional to the number of files,
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@874
PS2, Line 874: // during which time partial updates may be visible.
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@875
PS2, Line 875: // delete() and rename() can not guarantee atomicity. If the operations are interrupted,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@875
PS2, Line 875: // delete() and rename() can not guarantee atomicity. If the operations are interrupted,
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@876
PS2, Line 876: // the filesystem is left in an intermediate state. File owner and group are persisted,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@876
PS2, Line 876: // the filesystem is left in an intermediate state. File owner and group are persisted,
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@877
PS2, Line 877: // but the permissions model is not enforced. Authorization occurs at the level of the
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@877
PS2, Line 877: // but the permissions model is not enforced. Authorization occurs at the level of the
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/2/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:
http://gerrit.cloudera.org:8080/#/c/19165/2/testdata/bin/create-load-data.sh@140
PS2, Line 140: echo "metadata, but it cannot be loaded on isilon, s3, gcs, cos, oss or local and the"
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/2/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/19165/2/tests/custom_cluster/test_hdfs_fd_caching.py@138
PS2, Line 138: I
flake8: F821 undefined name 'IS_OSS'
http://gerrit.cloudera.org:8080/#/c/19165/2/tests/custom_cluster/test_hdfs_fd_caching.py@155
PS2, Line 155: I
flake8: F821 undefined name 'IS_OSS'
http://gerrit.cloudera.org:8080/#/c/19165/2/tests/custom_cluster/test_hdfs_fd_caching.py@206
PS2, Line 206: I
flake8: F821 undefined name 'IS_OSS'
http://gerrit.cloudera.org:8080/#/c/19165/2/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:
http://gerrit.cloudera.org:8080/#/c/19165/2/tests/custom_cluster/test_metastore_service.py@1205
PS2, Line 1205: I
flake8: F821 undefined name 'IS_OSS'
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 2
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 02:54:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19165/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/13//COMMIT_MSG@18
PS13, Line 18: a
nit. an
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 13
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 09 Nov 2022 00:27:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 11:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG@9
PS11, Line 9: Oobject
> nit. Object
Done
http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG@14
PS11, Line 14: Mdddanual ACID test verification.
> Could you elaborate a bit on how did you test on this feature? For example,
Done
http://gerrit.cloudera.org:8080/#/c/19165/11/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:
http://gerrit.cloudera.org:8080/#/c/19165/11/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@447
PS11, Line 447: a
> nit. an
Done
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 11
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Tue, 08 Nov 2022 09:20:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, lipenglin@sensorsdata.cn, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#16).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
This patch adds support for OSS (Aliyun Object Storage Service).
Using the hadoop-aliyun, the implementation is similar to other
remote FileSystems.
Tests:
- Prepare:
Initialize OSS-related environment variables:
OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT.
Compile and create hdfs test data on a ECS instance. Upload test data
to an OSS bucket.
- Modify all locations in HMS DB to point to the OSS bucket.
Remove some hdfs caching params. Run CORE tests.
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 91 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/16
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 16
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS15, Line 872: rename()
> AFAIK, e.g. s3(Amazon Simple Storage Service) neither does not support atom
Every rename request is converted into a copy + delete operation. Therefore, for the rename operation of an object (file), in the worst case, a large number of copied target objects will be generated, and finally the rename src file will be deleted.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 15
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 09 Nov 2022 12:08:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11820/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 16
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 09 Nov 2022 12:27:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#12).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
Add Aliyun OSS support This patch adds support for OSS (Aliyun Open Strorage Service).
Using the hadoop-aliyun, the implementation is similar to other remote FileSystems.
For more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service
Tests:
- Prepare:
Initialize OSS-related environment variables:
OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT.
Compile and create hdfs test data on a ECS instance. Upload test data
to a OSS bucket.
- Modify all locations in HMS DB to point to the OSS bucket.
Remove some hdfs caching params. Run CORE tests.
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 91 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/12
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 12
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19165/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/13//COMMIT_MSG@18
PS13, Line 18: a
> nit. an
Thanks for your quick review. I updated according to your comments, not sure if this version meets your requirements, not very sure if the commit msg content needs to be less than 72 is it for the description of the second paragraph or the whole commit msg?
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 13
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 09 Nov 2022 03:24:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 16: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 16
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 09 Nov 2022 18:02:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11697/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 4
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 03:29:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11741/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 8
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Mon, 31 Oct 2022 07:25:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#6).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
Add Aliyun OSS support This patch adds support for OSS (Aliyun Oobject Strorage Service).
Using the hadoop-aliyun, the implementation is similar to other remote FileSystems.
for more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service
Tests:
Run CORE tests.
Manual ACID test verification
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/planner/ExplainTest.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 93 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/6
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 6
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 1:
(14 comments)
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS1, Line 872: // Atomic operations: delete() and rename() are implemented by recursive
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@873
PS1, Line 873: // file-by-file operations. They take time at least proportional to the number of files,
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@874
PS1, Line 874: // during which time partial updates may be visible.
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@875
PS1, Line 875: // delete() and rename() can not guarantee atomicity. If the operations are interrupted,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@875
PS1, Line 875: // delete() and rename() can not guarantee atomicity. If the operations are interrupted,
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@876
PS1, Line 876: // the filesystem is left in an intermediate state. File owner and group are persisted,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@876
PS1, Line 876: // the filesystem is left in an intermediate state. File owner and group are persisted,
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@877
PS1, Line 877: // but the permissions model is not enforced. Authorization occurs at the level of the
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19165/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@877
PS1, Line 877: // but the permissions model is not enforced. Authorization occurs at the level of the
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/1/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:
http://gerrit.cloudera.org:8080/#/c/19165/1/testdata/bin/create-load-data.sh@140
PS1, Line 140: echo "metadata, but it cannot be loaded on isilon, s3, gcs, cos, oss or local and the"
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/1/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/19165/1/tests/custom_cluster/test_hdfs_fd_caching.py@138
PS1, Line 138: I
flake8: F821 undefined name 'IS_OSS'
http://gerrit.cloudera.org:8080/#/c/19165/1/tests/custom_cluster/test_hdfs_fd_caching.py@155
PS1, Line 155: I
flake8: F821 undefined name 'IS_OSS'
http://gerrit.cloudera.org:8080/#/c/19165/1/tests/custom_cluster/test_hdfs_fd_caching.py@206
PS1, Line 206: I
flake8: F821 undefined name 'IS_OSS'
http://gerrit.cloudera.org:8080/#/c/19165/1/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:
http://gerrit.cloudera.org:8080/#/c/19165/1/tests/custom_cluster/test_metastore_service.py@1205
PS1, Line 1205: I
flake8: F821 undefined name 'IS_OSS'
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 1
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 02:43:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#14).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
Add Aliyun OSS support This patch adds support for OSS (Aliyun Object Storage Service).
Using the hadoop-aliyun, the implementation is similar to other remote FileSystems.
Tests:
- Prepare:
Initialize OSS-related environment variables:
OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT.
Compile and create hdfs test data on a ECS instance. Upload test data
to an OSS bucket.
- Modify all locations in HMS DB to point to the OSS bucket.
Remove some hdfs caching params. Run CORE tests.
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 91 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/14
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 14
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 16:
(1 comment)
LGTM!
http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS15, Line 872: rename()
> Every rename request is converted into a copy + delete operation. Therefore
Done
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 16
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 09 Nov 2022 12:39:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 16:
(3 comments)
LGTM. I just have some minor comments.
http://gerrit.cloudera.org:8080/#/c/19165/16//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/16//COMMIT_MSG@20
PS16, Line 20: Run CORE tests
Just curious, are there any failed tests?
http://gerrit.cloudera.org:8080/#/c/19165/16/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/16/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS16, Line 872: Atomic operations: delete() and rename() are implemented by recursive
: // file-by-file operations. They take time at least proportional to the number of
: // files, during which time partial updates may be visible. delete() and rename()
: // can not guarantee atomicity. If the operations are interrupted,
: // the filesystem is left in an intermediate state.
This part seems unrelated to read/write permissions. Can we move them to somewhere else? If there are no codes related to these, we can put them in the commit message and skip.py:
https://github.com/apache/impala/blob/97a506c6560e7a0c6c5ba7d098f07c6b04ef6cad/tests/common/skip.py#L85
http://gerrit.cloudera.org:8080/#/c/19165/16/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@879
PS16, Line 879: Directory last access time is not tracked.
: // The append operation is not supported.
seems unrelated too
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 16
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Mon, 14 Nov 2022 06:47:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#4).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/planner/ExplainTest.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
15 files changed, 92 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/4
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 4
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19165/4/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:
http://gerrit.cloudera.org:8080/#/c/19165/4/testdata/bin/create-load-data.sh@140
PS4, Line 140: echo "metadata, but it cannot be loaded on isilon, s3, gcs, cos, oss or local and the"
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19165/4/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:
http://gerrit.cloudera.org:8080/#/c/19165/4/tests/custom_cluster/test_metastore_service.py@1205
PS4, Line 1205: I
flake8: F821 undefined name 'IS_OSS'
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 4
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 03:15:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#13).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
Add Aliyun OSS support This patch adds support for OSS (Aliyun Object Storage Service).
Using the hadoop-aliyun, the implementation is similar to other remote FileSystems.
For more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service
Tests:
- Prepare:
Initialize OSS-related environment variables:
OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT.
Compile and create hdfs test data on a ECS instance. Upload test data
to a OSS bucket.
- Modify all locations in HMS DB to point to the OSS bucket.
Remove some hdfs caching params. Run CORE tests.
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 91 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/13
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 13
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 5:
(3 comments)
Thanks for the patch. It looks good to me, just some minor questions.
http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG@7
PS5, Line 7: Support Aliyun OSS File System
Would be good to have a detailed commit message, including the description of the patch, the added or removed flags if involved, and the tests you have ran.
http://gerrit.cloudera.org:8080/#/c/19165/5/be/src/runtime/io/disk-io-mgr-test.cc
File be/src/runtime/io/disk-io-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/19165/5/be/src/runtime/io/disk-io-mgr-test.cc@1724
PS5, Line 1724: FLAGS_num_oss_io_threads
Testcase failed at DiskIoMgrTest::VerifyNumThreadsParameter, https://jenkins.impala.io/job/pre-review-test/1466/, ran with be test only, could have other failures. Would be good to run a preview test before review: https://jenkins.impala.io/job/pre-review-test/
http://gerrit.cloudera.org:8080/#/c/19165/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS5, Line 872: // Atomic operations: delete() and rename() are implemented by recursive
: // file-by-file operations.
It is good to have such explanations for the atomicity of OSS. But not sure if my understanding is correct, from the document of Hadoop https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/introduction.html#Core_Expectations_of_a_Hadoop_Compatible_FileSystem, for a Hadoop-compatible FileSystem, the deletion of a directory doesn't need to be atomic, but the rename of a directory must be atomic, would be good if you could separate the case of atomic operations on a file and a directory for OSS.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 5
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 25 Oct 2022 16:46:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 18:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11857/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 18
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 16 Nov 2022 05:21:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 17:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19165/16/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/16/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS16, Line 872: File owner and group are
: // persisted, but the permissions model is not enforced. Authorization occurs at
: // the level of the entire Aliyun account via Aliyun Resource Access Management.
: // The append operation is not supported.
: if (FileSystemUtil.isOSSFileSystem(fs)) return true;
> This part seems unrelated to read/write permissions. Can we move them to so
Done
http://gerrit.cloudera.org:8080/#/c/19165/16/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@879
PS16, Line 879:
: /**
> seems unrelated too
the append operation is related about write. so i still keep comment here.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 17
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 16 Nov 2022 03:45:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS15, Line 872: rename()
> Done
who can help with +2 & merge? could you help with that?
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 15
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Thu, 10 Nov 2022 03:59:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#8).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
This patch adds support for OSS (Aliyun Oobject Strorage Service).
Using the hadoop-aliyun, the implementation is similar to other remote FileSystems.
For more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service
Tests:
Run CORE tests.
Mdddanual ACID test verification.
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/planner/ExplainTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M testdata/workloads/functional-planner/queries/PlannerTest/scan-node-fs-scheme.test
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
17 files changed, 113 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/8
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 8
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 15:
(2 comments)
Thanks for this implementing, it will be very helpful!
Just left some minor comments.
http://gerrit.cloudera.org:8080/#/c/19165/15//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/15//COMMIT_MSG@9
PS15, Line 9: to
nit: each line should be limited to 72 characters or less if possible.
https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala
http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS15, Line 872: rename()
AFAIK, e.g. s3(Amazon Simple Storage Service) neither does not support atomic rename operations. I just want to know what the unpredictable and terrible situation will occur when multiple clients rename a file at the same time?
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 15
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 09 Nov 2022 10:06:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 16:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19165/16//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19165/16//COMMIT_MSG@20
PS16, Line 20: Run CORE tests
> Just curious, are there any failed tests?
i just running ./bin/run-all-tests.sh, and seems everything is ok
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 16
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 16 Nov 2022 03:55:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
This patch adds support for OSS (Aliyun Object Storage Service).
Using the hadoop-aliyun, the implementation is similar to other
remote FileSystems.
Tests:
- Prepare:
Initialize OSS-related environment variables:
OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT.
Compile and create hdfs test data on a ECS instance. Upload test data
to an OSS bucket.
- Modify all locations in HMS DB to point to the OSS bucket.
Remove some hdfs caching params. Run CORE tests.
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Reviewed-on: http://gerrit.cloudera.org:8080/19165
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 87 insertions(+), 17 deletions(-)
Approvals:
Quanlong Huang: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 19
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 )
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
Patch Set 18: Code-Review+2
(1 comment)
Carry +1 from Yida.
http://gerrit.cloudera.org:8080/#/c/19165/17/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/19165/17/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872
PS17, Line 872: // enforced. Authorization
reformatted this in patch set 18.
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 18
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Comment-Date: Wed, 16 Nov 2022 05:03:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Posted by "hezhenhua.hzh (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Yida Wu, lipenglin@sensorsdata.cn, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19165
to look at the new patch set (#17).
Change subject: IMPALA-11683: Support Aliyun OSS File System
......................................................................
IMPALA-11683: Support Aliyun OSS File System
This patch adds support for OSS (Aliyun Object Storage Service).
Using the hadoop-aliyun, the implementation is similar to other
remote FileSystems.
Tests:
- Prepare:
Initialize OSS-related environment variables:
OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT.
Compile and create hdfs test data on a ECS instance. Upload test data
to an OSS bucket.
- Modify all locations in HMS DB to point to the OSS bucket.
Remove some hdfs caching params. Run CORE tests.
Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
---
M be/src/exec/hdfs-table-sink.cc
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/testutil/TestUtils.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 88 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/17
--
To view, visit http://gerrit.cloudera.org:8080/19165
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910
Gerrit-Change-Number: 19165
Gerrit-PatchSet: 17
Gerrit-Owner: hezhenhua.hzh <he...@alibaba-inc.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: hezhenhua.hzh <he...@alibaba-inc.com>