You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Xiang Yang (Code Review)" <ge...@cloudera.org> on 2022/10/17 14:49:47 UTC

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Xiang Yang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19110


Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS(Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

Testing:
 - Manually create | insert data to | query data from | drop an iceberg
   table stored on OBS FileSystem.
 - Ran core jobs.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
7 files changed, 66 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/19110/3
-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 3
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19110

to look at the new patch set (#4).

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

WIP IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS(Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

New flags for OBS:
- num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16.

Testing:
 - Manually create | insert data to | query data from | drop an iceberg
   table stored on OBS FileSystem.
 - Ran core jobs.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
M java/pom.xml
9 files changed, 77 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/19110/4
-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 4
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 3:

(3 comments)

Thank you for the change. Looks good to me. Only some minor comments.

http://gerrit.cloudera.org:8080/#/c/19110/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19110/3//COMMIT_MSG@10
PS3, Line 10: The implementation is similar to other remote FileSystems.
Would be good to also mention the new added start option num_obs_io_threads in the commit message.


http://gerrit.cloudera.org:8080/#/c/19110/3//COMMIT_MSG@12
PS3, Line 12: Testing:
Do we need to consider running existing testcases or test data using the huawei OBS as an option?


http://gerrit.cloudera.org:8080/#/c/19110/3/java/executor-deps/pom.xml
File java/executor-deps/pom.xml:

http://gerrit.cloudera.org:8080/#/c/19110/3/java/executor-deps/pom.xml@226
PS3, Line 226: <version>3.1.1-hw-42</version>
Would be better to have a variable for version control like ${cos.version} and ${ozone.version}.



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 3
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 17 Oct 2022 16:38:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 10: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 10
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 05 Feb 2023 20:14:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12219/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 8
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 23 Jan 2023 06:40:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9024/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 10
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 05 Feb 2023 08:57:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS (Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

New flags for OBS:
- num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16.

Testing:
 - Upload hdfs test data to an OBS bucket. Modify all locations in HMS
   DB to point to the OBS bucket. Remove some hdfs caching params.
   Run CORE tests.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Reviewed-on: http://gerrit.cloudera.org:8080/19110
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
15 files changed, 117 insertions(+), 18 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 13
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11643/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 3
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Oct 2022 15:10:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 6:

(2 comments)

> Patch Set 6:
> 
> Is the patch ready for review or still WIP?

http://gerrit.cloudera.org:8080/#/c/19110/5/be/src/runtime/io/disk-io-mgr.cc
File be/src/runtime/io/disk-io-mgr.cc:

http://gerrit.cloudera.org:8080/#/c/19110/5/be/src/runtime/io/disk-io-mgr.cc@148
PS5, Line 148: 16
> If this is not an empirical value, we can add a comment: 'TODO: choose the 
just follow other filesystem's corresponding value :) users can change it according to there own cluster's status.


http://gerrit.cloudera.org:8080/#/c/19110/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:

http://gerrit.cloudera.org:8080/#/c/19110/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@458
PS5, Line 458: e =
> nit: if
I think 'iff' is the meaning of 'if and only if', but a typo.



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 6
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 02:32:03 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 11: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 11
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 07 Feb 2023 18:05:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 4:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/19110/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19110/3//COMMIT_MSG@10
PS3, Line 10: The implementation is similar to other remote FileSystems.
> Would be good to also mention the new added start option num_obs_io_threads
Done


http://gerrit.cloudera.org:8080/#/c/19110/3//COMMIT_MSG@12
PS3, Line 12: New flag
> Do we need to consider running existing testcases or test data using the hu
seems a bit complicated, I'll try it, but may take some days.


http://gerrit.cloudera.org:8080/#/c/19110/3/java/executor-deps/pom.xml
File java/executor-deps/pom.xml:

http://gerrit.cloudera.org:8080/#/c/19110/3/java/executor-deps/pom.xml@226
PS3, Line 226: in>
> Would be better to have a variable for version control like ${cos.version} 
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 4
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 19 Oct 2022 09:06:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 11:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/19110/10/tests/common/impala_test_suite.py
File tests/common/impala_test_suite.py:

http://gerrit.cloudera.org:8080/#/c/19110/10/tests/common/impala_test_suite.py@1074
PS10, Line 1074:         in ['s3', 'isilon', 'local', 'abfs', 'adls', 'gs', 'cosn', 'ozone', 'obs']:
> I think you want to add obs to this list to address your hbase test failure
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 11
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 07 Feb 2023 03:49:13 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9025/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 10
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 05 Feb 2023 15:09:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, lipenglin@apache.org, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19110

to look at the new patch set (#7).

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

WIP IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS(Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

New flags for OBS:
- num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16.

Testing:
 - Upload hdfs test data to a OBS bucket. Modify all locations in HMS
   DB to point to the OBS bucket. Remove some hdfs caching params.
   Run CORE tests.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 118 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/19110/7
-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 7
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, lipenglin@apache.org, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19110

to look at the new patch set (#8).

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

WIP IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS(Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

New flags for OBS:
- num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16.

Testing:
 - Upload hdfs test data to a OBS bucket. Modify all locations in HMS
   DB to point to the OBS bucket. Remove some hdfs caching params.
   Run CORE tests.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
14 files changed, 104 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/19110/8
-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 8
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 10: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9024/


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 10
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 05 Feb 2023 14:09:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, lipenglin@sensorsdata.cn, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19110

to look at the new patch set (#6).

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

WIP IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS(Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

New flags for OBS:
- num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16.

Testing:
 - Manually create | insert data to | query data from | drop an iceberg
   table stored on OBS FileSystem.
 - Ran core jobs.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
M java/pom.xml
9 files changed, 78 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/19110/6
-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 6
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11662/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 4
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 19 Oct 2022 09:16:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 6:

Is the patch ready for review or still WIP?


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 6
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 19:12:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 6:

unfortunately not ready. I waste time on fixing my dev environment and other work, this week I'm starting to upload testdata to a obs instance and run core test. Sorry for waiting so long.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 6
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 02:39:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 10: -Code-Review

(1 comment)

http://gerrit.cloudera.org:8080/#/c/19110/10/tests/common/impala_test_suite.py
File tests/common/impala_test_suite.py:

http://gerrit.cloudera.org:8080/#/c/19110/10/tests/common/impala_test_suite.py@1074
PS10, Line 1074:         in ['s3', 'isilon', 'local', 'abfs', 'adls', 'gs', 'cosn', 'ozone']:
I think you want to add obs to this list to address your hbase test failures.



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 10
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 06 Feb 2023 21:21:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12253/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 9
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 29 Jan 2023 09:17:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 12:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9038/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 12
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 09 Feb 2023 02:53:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 12: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 12
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 09 Feb 2023 02:53:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 10: Code-Review+1

> Patch Set 10:
> 
> (3 comments)
> 
> Thanks Michael!
> I run the core test based on patch10 with only specified table-formats:
> 
> export BE_TEST=false
> export FE_TEST=false
> export JDBC_TEST=false
> export RUN_TESTS_ARGS="--table_formats=parquet/none,kudu/none"
> time bin/run-all-tests.sh -e core
> 
> 
> the hdfs ee test log: https://issues.apache.org/jira/secure/attachment/13055151/hdfs_ee_test_patch10.log
> the obs ee test log: https://issues.apache.org/jira/secure/attachment/13055152/obs_ee_test_patch10.log
> 
> When test HDFS filesystem, the HBase services are stopped.
> When test OBS filesystem, the HBase and HDFS services are stopped.
> Most of failed cases were failed because of hbase tables.
> Some OBS failed cases can success by running with './bin/impala-py.test ...' manually.

The changes all look right to me now. I'd have to spend some time sifting through the test results; there are some oddities like test_hbase_inserts fails with OBS but is skipped with HDFS.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 10
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 06 Feb 2023 17:15:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 8:

(2 comments)

Hi yida, I do the ee test as described in commit message, following quanlong's blog: https://blog.csdn.net/huang_quanlong/article/details/114937678.
This is the result: https://issues.apache.org/jira/secure/attachment/13054770/obs_ee_test_core_results.csv 
and the hdfs ee test log: https://issues.apache.org/jira/secure/attachment/13054772/hdfs_ee_test.log
and the obs ee test log: https://issues.apache.org/jira/secure/attachment/13054771/obs_ee_test.log

I found there are some failed cases, such as 'tests/data_errors/test_data_errors.py::TestAvroErrors::test_avro_errors',  failed because of the '$NAMENODE' variable. May be we can replace it to '$DEFAULT_FS'.
Another failed case is './bin/impala-py.test tests/query_test/test_iceberg.py::TestIcebergTable::test_load', maybe because of the random generated database's location is defaulted to HDFS.
Do you think I should solve these problems in this patch, or create a new jira issue?
CC quanlong, thanks for review this.

http://gerrit.cloudera.org:8080/#/c/19110/8/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/19110/8/tests/custom_cluster/test_metastore_service.py@1176
PS8, Line 1176:         if IS_HDFS:
Hi yida, quanlong, do you accept this simplify method?


http://gerrit.cloudera.org:8080/#/c/19110/8/tests/custom_cluster/test_metastore_service.py@1204
PS8, Line 1204:       if IS_HDFS:
Hi yida, quanlong, same as above



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 8
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 23 Jan 2023 06:42:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 5:

(2 comments)

LGTM. Left some minor comments.

http://gerrit.cloudera.org:8080/#/c/19110/5/be/src/runtime/io/disk-io-mgr.cc
File be/src/runtime/io/disk-io-mgr.cc:

http://gerrit.cloudera.org:8080/#/c/19110/5/be/src/runtime/io/disk-io-mgr.cc@148
PS5, Line 148: 16
If this is not an empirical value, we can add a comment: 'TODO: choose the default empirically.'


http://gerrit.cloudera.org:8080/#/c/19110/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:

http://gerrit.cloudera.org:8080/#/c/19110/5/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@458
PS5, Line 458: iff
nit: if
We should also change the 'iff' -> 'if' in the comments above.



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 5
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 01 Dec 2022 03:19:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 10:

(3 comments)

Thanks Michael!
I run the core test based on patch10 with only specified table-formats:

export BE_TEST=false
export FE_TEST=false
export JDBC_TEST=false
export RUN_TESTS_ARGS="--table_formats=parquet/none,kudu/none"
time bin/run-all-tests.sh -e core


the hdfs ee test log: https://issues.apache.org/jira/secure/attachment/13055151/hdfs_ee_test_patch10.log
the obs ee test log: https://issues.apache.org/jira/secure/attachment/13055152/obs_ee_test_patch10.log

When test HDFS filesystem, the HBase services are stopped.
When test OBS filesystem, the HBase and HDFS services are stopped.
Most of failed cases were failed because of hbase tables.
Some OBS failed cases can success by running with './bin/impala-py.test ...' manually.

http://gerrit.cloudera.org:8080/#/c/19110/9/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/19110/9/bin/impala-config.sh@710
PS9, Line 710:   export OBS_ACCESS_KEY OBS_SECRET_KEY OBS_ENDPOINT DEFAULT_FS ENABLE_OBS_FILESYSTEM=true
> This shouldn't need to be set. It's only used if your warehouse is not "obs
Done


http://gerrit.cloudera.org:8080/#/c/19110/9/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/19110/9/tests/custom_cluster/test_metastore_service.py@1214
PS9, Line 1214:         if IS_HDFS or IS_OZONE:
> This should be IS_HDFS or IS_OZONE now. Probably should have updated the co
Done


http://gerrit.cloudera.org:8080/#/c/19110/9/tests/custom_cluster/test_metastore_service.py@1242
PS9, Line 1242:       if IS_HDFS or IS_OZONE:
> Same as above.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 10
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 05 Feb 2023 08:27:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Yida Wu, lipenglin@apache.org, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19110

to look at the new patch set (#10).

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS (Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

New flags for OBS:
- num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16.

Testing:
 - Upload hdfs test data to an OBS bucket. Modify all locations in HMS
   DB to point to the OBS bucket. Remove some hdfs caching params.
   Run CORE tests.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
15 files changed, 116 insertions(+), 17 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/19110/10
-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 10
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12313/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 10
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 05 Feb 2023 08:24:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12320/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 11
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 07 Feb 2023 03:58:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12218/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 7
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 23 Jan 2023 06:22:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Yida Wu, lipenglin@apache.org, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19110

to look at the new patch set (#9).

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS (Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

New flags for OBS:
- num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16.

Testing:
 - Upload hdfs test data to an OBS bucket. Modify all locations in HMS
   DB to point to the OBS bucket. Remove some hdfs caching params.
   Run CORE tests.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
15 files changed, 115 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/19110/9
-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 9
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 9:

> Uploaded patch set 9.

Hi all, Patch#9 add OBS related configurations to core-site.xml to reduce manual work, and export 'WAREHOUSE_LOCATION_PREFIX' env variable.

I figured out that when the 'fs.defaultFS' in core-site.xml set to 'obs://...', the table's location will defaulted to 'obs://...' but 'hdfs://localhost:20500', then some failed cases in Patch#8 will be passed, thus the '$NAMENODE' variable I mentioned doesn't need to change, see https://github.com/apache/impala/blob/4.2.0/tests/util/hdfs_util.py#L55 .

Another challenge I face is some EE tests rely on hbase, but hbase table store data on hdfs currently, so I need to start the hdfs cluster. But namenode will fail to start when 'fs.defaultFS' set to 'obs://...'. So I need to initialize 'fs.defaultFS' to 'hdfs://localhost:20500' and start hdfs cluster, then change it to 'obs://...' to start hive and impala cluster. It'll then cause some EE tests failed because of unique_database or table location defaults to 'hdfs://localhost:20500/...' for unknown reason. So I export 'WAREHOUSE_LOCATION_PREFIX' env variable so the 'hive.metastore.warehouse.dir' and 'hive.metastore.warehouse.external.dir' in hive-site.xml fixed with explicit filesystem schema, or I have to research how to modify hbase table's location to point to obs filesystem.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 9
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 29 Jan 2023 09:56:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 9:

(3 comments)

> Patch Set 9:
> 
> > Uploaded patch set 9.
> 
> Hi all, Patch#9 add OBS related configurations to core-site.xml to reduce manual work, and export 'WAREHOUSE_LOCATION_PREFIX' env variable.
> 
> I figured out that when the 'fs.defaultFS' in core-site.xml set to 'obs://...', the table's location will defaulted to 'obs://...' but 'hdfs://localhost:20500', then some failed cases in Patch#8 will be passed, thus the '$NAMENODE' variable I mentioned doesn't need to change, see https://github.com/apache/impala/blob/4.2.0/tests/util/hdfs_util.py#L55 .
> 
> Another challenge I face is some EE tests rely on hbase, but hbase table store data on hdfs currently, so I need to start the hdfs cluster. But namenode will fail to start when 'fs.defaultFS' set to 'obs://...'. So I need to initialize 'fs.defaultFS' to 'hdfs://localhost:20500' and start hdfs cluster, then change it to 'obs://...' to start hive and impala cluster. It'll then cause some EE tests failed because of unique_database or table location defaults to 'hdfs://localhost:20500/...' for unknown reason. So I export 'WAREHOUSE_LOCATION_PREFIX' env variable so the 'hive.metastore.warehouse.dir' and 'hive.metastore.warehouse.external.dir' in hive-site.xml fixed with explicit filesystem schema, or I have to research how to modify hbase table's location to point to obs filesystem.

Tests that use hbase should be skipped via https://github.com/apache/impala/blob/master/tests/common/skip.py#L78, and hbase shouldn't be started - https://github.com/apache/impala/blob/master/testdata/bin/run-all.sh#L62 - unless DEFAULT_FS == 'hdfs://localhost:20500'.

http://gerrit.cloudera.org:8080/#/c/19110/9/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/19110/9/bin/impala-config.sh@710
PS9, Line 710:   WAREHOUSE_LOCATION_PREFIX=${DEFAULT_FS}
This shouldn't need to be set. It's only used if your warehouse is not "obs://${OBS_BUCKET}/test-warehouse".


http://gerrit.cloudera.org:8080/#/c/19110/9/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/19110/9/tests/custom_cluster/test_metastore_service.py@1214
PS9, Line 1214:         if IS_HDFS:
This should be IS_HDFS or IS_OZONE now. Probably should have updated the comment.


http://gerrit.cloudera.org:8080/#/c/19110/9/tests/custom_cluster/test_metastore_service.py@1242
PS9, Line 1242:       if IS_HDFS:
Same as above.



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 9
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 30 Jan 2023 17:47:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19110

to look at the new patch set (#5).

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

WIP IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS(Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

New flags for OBS:
- num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16.

Testing:
 - Manually create | insert data to | query data from | drop an iceberg
   table stored on OBS FileSystem.
 - Ran core jobs.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
M java/pom.xml
9 files changed, 78 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/19110/5
-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 5
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11909/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 5
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 27 Nov 2022 09:00:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12064/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 6
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 25 Dec 2022 09:58:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 12: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 12
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 09 Feb 2023 08:10:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 11: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 11
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 09 Feb 2023 02:52:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Yida Wu, lipenglin@apache.org, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19110

to look at the new patch set (#11).

Change subject: IMPALA-11629: Support for huawei OBS FileSystem
......................................................................

IMPALA-11629: Support for huawei OBS FileSystem

This patch adds support for huawei OBS (Object Storage Service)
FileSystem. The implementation is similar to other remote FileSystems.

New flags for OBS:
- num_obs_io_threads: Number of OBS I/O threads. Defaults to be 16.

Testing:
 - Upload hdfs test data to an OBS bucket. Modify all locations in HMS
   DB to point to the OBS bucket. Remove some hdfs caching params.
   Run CORE tests.

Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
---
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M java/executor-deps/pom.xml
M java/pom.xml
M testdata/bin/create-load-data.sh
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_metastore_service.py
M tests/util/filesystem_utils.py
15 files changed, 117 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/19110/11
-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 11
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 6: Code-Review+1

LGTM


-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 6
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 02:56:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-11629: Support for huawei OBS FileSystem

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )

Change subject: WIP IMPALA-11629: Support for huawei OBS FileSystem
......................................................................


Patch Set 7:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/19110/7/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/19110/7/bin/impala-config.sh@696
PS7, Line 696:   export OBS_ACCESS_KEY="${OBS_ACCESS_KEY:?OBS_ACCESS_KEY cannot be an empty string for OBS}"
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/19110/7/bin/impala-config.sh@697
PS7, Line 697:   export OBS_SECRET_KEY="${OBS_SECRET_KEY:?OBS_SECRET_KEY cannot be an empty string for OBS}"
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/19110/7/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:

http://gerrit.cloudera.org:8080/#/c/19110/7/testdata/bin/create-load-data.sh@122
PS7, Line 122: SCHEMA_MISMATCH_ERROR+="but it cannot be loaded on Isilon, s3, gcs, cos, oss, obs or local "
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/19110/7/tests/common/skip.py
File tests/common/skip.py:

http://gerrit.cloudera.org:8080/#/c/19110/7/tests/common/skip.py@83
PS7, Line 83: S
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19110/7/tests/util/filesystem_utils.py
File tests/util/filesystem_utils.py:

http://gerrit.cloudera.org:8080/#/c/19110/7/tests/util/filesystem_utils.py@65
PS7, Line 65: '
flake8: E501 line too long (103 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/19110
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1
Gerrit-Change-Number: 19110
Gerrit-PatchSet: 7
Gerrit-Owner: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Anonymous Coward <li...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 23 Jan 2023 06:02:19 +0000
Gerrit-HasComments: Yes