You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sahil Takiar (Code Review)" <ge...@cloudera.org> on 2020/03/02 15:09:39 UTC
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15334
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
IMPALA-9400: Initial Ozone support
Ozone is a block-based filesystem store and is similar to HDFS in
many aspects. This patch adds initial support for reading data stored
on Ozone and writing data to Ozone. It adds Ozone to the whitelist of
Impala writable filesystems and adds the necessary Ozone client jars
to the Impala classpath.
Rather than updating to a new CDP GBN with the added Ozone client jars,
I just modified the existing GBN to include the new jars in the Maven
repo.
Testing:
* Manually tested that connecting to an external Ozone cluster works and
that Impala can create an Ozone-backed table, write to the
table, read from the table, and drop the table
* Ran core tests with and without USE_CDP_HIVE=true
Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
---
M be/src/exec/hdfs-table-sink.cc
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/pom.xml
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M impala-parent/pom.xml
7 files changed, 55 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/15334/1
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5383/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2020 15:54:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18
PS1, Line 18:
> From what I know so far, Ozone does expose block locations.
I don't want to scope creep this, but another thought I had was that it seems weird to allow Ozone by default but not allow HDFS erasure coding (ALLOW_ERASURE_CODED_FILES=false). Maybe we should just leave as-is though... might be worth thinking about flipping the result.
Treating it as a remote FS by default makes sense to me, I think, since we then avoid the overhead of fetching and storing the block locations.
No we don't support rack-aware scheduling in Impala..
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2020 18:27:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5449/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2020 03:04:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18
PS1, Line 18:
I have some questions about the block locations and scheduling. Does this load block locations and do locality-based scheduling?
It looks like there isn't any code added to DiskIoMgr to route the reads to a particular queue - it seems like in DiskIoMgr::AssignQueue() it falls through in a weird way.
OK if we want to defer this but should probably file a JIRA.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2020 17:02:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18
PS1, Line 18:
> I think this is fine to leave enabled by default. This would be an argument
Sounds good. Will follow up on the EC support separately.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2020 02:20:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18
PS1, Line 18:
> I have some questions about the block locations and scheduling. Does this l
From what I know so far, Ozone does expose block locations.
This patch changes supportsStorageIds so that it returns true for Ozone filesystems as well (this should make the HdfsScanNode fetch the block locations and perform scheduling based on the block locations). However, I think I might revert that part of the patch.
I believe, for the most part, we expect to deploy Ozone on a dedicated storage cluster that is separate from the Impala compute cluster - e.g. separate compute vs storage cluster. So I think, at least for now, its okay if we treat Ozone as remote storage (similar to what we do for ABFS and S3A).
Even with separate storage + compute clusters, block locations might help with rack-aware scheduling, but I don't think we support that in Impala yet?
I'll make the fixes for the DiskIoMgr.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2020 17:37:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5452/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2020 17:57:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Anonymous Coward (572), Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15334
to look at the new patch set (#4).
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
IMPALA-9400: Initial Ozone support
Ozone is a block-based filesystem store and is similar to HDFS in
many aspects. This patch adds initial support for reading data stored
on Ozone and writing data to Ozone. It adds Ozone to the whitelist of
Impala writable filesystems and adds the necessary Ozone client jars
to the Impala classpath.
Initially, Ozone will be treated as a remote storage service, similar to
how Impala treats S3 and ABFS. Ozone will typically be deployed on a
separate storage cluster, so all reads will be remote.
Since remote HDFS and Ozone are separate storage systems, with distinct
performance characteristics, they are treated as separate systems for
disk scheduling purposes.
Rather than updating to a new CDP GBN with the added Ozone client jars,
I just modified the existing GBN to include the new jars in the Maven
repo.
Testing:
* Manually tested that connecting to an external Ozone cluster works and
that Impala can create an Ozone-backed table, write to the
table, read from the table, and drop the table
* Ran core tests with and without USE_CDP_HIVE=true
Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
---
M be/src/exec/hdfs-table-sink.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/pom.xml
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M impala-parent/pom.xml
10 files changed, 75 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/15334/4
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 2: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15334/2/be/src/runtime/io/disk-io-mgr.cc
File be/src/runtime/io/disk-io-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/15334/2/be/src/runtime/io/disk-io-mgr.cc@113
PS2, Line 113: DEFINE_int32(num_ozone_io_threads, 16, "Number of Ozone I/O threads");
Can you mention in the commit message that it's treated as separate from remote HDFS for disk scheduling purposes and maybe the rationale (if any - there doesn't need to be, but I'd guess it makes sense if you think that it's probably a separate deployment from HDFS and has different perf characteristics).
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2020 21:43:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5412/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2020 03:47:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5420/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2020 18:41:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18
PS1, Line 18:
> I don't want to scope creep this, but another thought I had was that it see
Do you mean add the flag ALLOW_OZONE_FILES and set it to false by default, or do you mean set ALLOW_ERASURE_CODED_FILES=true by default?
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2020 18:38:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Anonymous Coward (572), Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15334
to look at the new patch set (#2).
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
IMPALA-9400: Initial Ozone support
Ozone is a block-based filesystem store and is similar to HDFS in
many aspects. This patch adds initial support for reading data stored
on Ozone and writing data to Ozone. It adds Ozone to the whitelist of
Impala writable filesystems and adds the necessary Ozone client jars
to the Impala classpath.
Initially, Ozone will be treated as a remote storage service, similar to
how Impala treats S3 and ABFS. Ozone will typically be deployed on a
separate storage cluster, so all reads will be remote.
Rather than updating to a new CDP GBN with the added Ozone client jars,
I just modified the existing GBN to include the new jars in the Maven
repo.
Testing:
* Manually tested that connecting to an external Ozone cluster works and
that Impala can create an Ozone-backed table, write to the
table, read from the table, and drop the table
* Ran core tests with and without USE_CDP_HIVE=true
Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
---
M be/src/exec/hdfs-table-sink.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/pom.xml
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M impala-parent/pom.xml
9 files changed, 72 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/15334/2
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Anonymous Coward (572) has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 1:
Ozone does support block locations (LocatedFIleStatus) and we have tested it against Hive. It does not however support short circuit reads.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2020 18:03:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5449/
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2020 07:38:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18
PS1, Line 18:
> Do you mean add the flag ALLOW_OZONE_FILES and set it to false by default,
I think this is fine to leave enabled by default. This would be an argument for enabling erasure encoding.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2020 19:15:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
IMPALA-9400: Initial Ozone support
Ozone is a block-based filesystem store and is similar to HDFS in
many aspects. This patch adds initial support for reading data stored
on Ozone and writing data to Ozone. It adds Ozone to the whitelist of
Impala writable filesystems and adds the necessary Ozone client jars
to the Impala classpath.
Initially, Ozone will be treated as a remote storage service, similar to
how Impala treats S3 and ABFS. Ozone will typically be deployed on a
separate storage cluster, so all reads will be remote.
Since remote HDFS and Ozone are separate storage systems, with distinct
performance characteristics, they are treated as separate systems for
disk scheduling purposes.
Rather than updating to a new CDP GBN with the added Ozone client jars,
I just modified the existing GBN to include the new jars in the Maven
repo.
Testing:
* Manually tested that connecting to an external Ozone cluster works and
that Impala can create an Ozone-backed table, write to the
table, read from the table, and drop the table
* Ran core tests with and without USE_CDP_HIVE=true
Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Reviewed-on: http://gerrit.cloudera.org:8080/15334
Reviewed-by: Sahil Takiar <st...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-table-sink.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/pom.xml
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M impala-parent/pom.xml
10 files changed, 75 insertions(+), 5 deletions(-)
Approvals:
Sahil Takiar: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 5
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2020 22:53:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5396/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2020 03:05:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Anonymous Coward (572), Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15334
to look at the new patch set (#3).
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
IMPALA-9400: Initial Ozone support
Ozone is a block-based filesystem store and is similar to HDFS in
many aspects. This patch adds initial support for reading data stored
on Ozone and writing data to Ozone. It adds Ozone to the whitelist of
Impala writable filesystems and adds the necessary Ozone client jars
to the Impala classpath.
Initially, Ozone will be treated as a remote storage service, similar to
how Impala treats S3 and ABFS. Ozone will typically be deployed on a
separate storage cluster, so all reads will be remote.
Since remote HDFS and Ozone are separate storage systems, with distinct
performance characteristics, they are treated as separate systems for
disk scheduling purposes.
Rather than updating to a new CDP GBN with the added Ozone client jars,
I just modified the existing GBN to include the new jars in the Maven
repo.
Testing:
* Manually tested that connecting to an external Ozone cluster works and
that Impala can create an Ozone-backed table, write to the
table, read from the table, and drop the table
* Ran core tests with and without USE_CDP_HIVE=true
Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
---
M be/src/exec/hdfs-table-sink.cc
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/util/hdfs-util.cc
M be/src/util/hdfs-util.h
M bin/impala-config.sh
M fe/pom.xml
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M impala-parent/pom.xml
9 files changed, 72 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/15334/3
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 )
Change subject: IMPALA-9400: Initial Ozone support
......................................................................
Patch Set 4: Code-Review+2
Carrying +2. Made a simple fix for disk-io-mgr-test.cc.
--
To view, visit http://gerrit.cloudera.org:8080/15334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99
Gerrit-Change-Number: 15334
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (572)
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2020 17:56:56 +0000
Gerrit-HasComments: No