You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2023/01/23 23:17:25 UTC

[Impala-ASF-CR] IMPALA-11476: Support Ozone erasure coding

Hello Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19324

to look at the new patch set (#10).

Change subject: IMPALA-11476: Support Ozone erasure coding
......................................................................

IMPALA-11476: Support Ozone erasure coding

Adds support for identifying erasure coding policy with Ozone. Enables
testing Ozone with erasure coding.

Test updates:
- test_exclusive_coordinator_plan: Ozone+EC blocks are 768MB, which is
  larger than all tables in our test environment. Use tpch_parquet which
  we rely on having 3 files (by loading from snapshot in this case).
- test_new_file_shorter: receives an EOFException when seeking with EC
- test_local_read: erasure-coded-bytes-read is also tied to IMPALA-11697
- test_erasure_coding: Ozone doesn't report files as erasure-coded
  (HDDS-7603)

Testing:
- Passes core E2E and custom cluster tests with TARGET_FILESYSTEM=ozone
  and ERASURE_CODING=true.

Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1
---
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java
M testdata/cluster/admin
M testdata/cluster/node_templates/common/etc/hadoop/conf/ozone-site.xml.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/custom_cluster/test_coordinators.py
M tests/query_test/test_io_metrics.py
M tests/query_test/test_scanners.py
10 files changed, 103 insertions(+), 35 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/19324/10
-- 
To view, visit http://gerrit.cloudera.org:8080/19324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1
Gerrit-Change-Number: 19324
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>