[jira] [Commented] (IMPALA-11476) Support for Ozone with Erasure Coding

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680733#comment-17680733 ] 

ASF subversion and git services commented on IMPALA-11476:
----------------------------------------------------------

Commit bbb0b4939dc8791b9b505f7bff46ce411b31ae6d in impala's branch refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=bbb0b4939 ]

IMPALA-11476: Support Ozone erasure coding

Adds support for identifying erasure coding policy with Ozone. Enables
testing Ozone with erasure coding.

Omits support for identifying erasure coding policy with the o3fs
protocol as that protocol is effectively deprecated and its classes
don't provide access to the ObjectStore.

Refactors volumeBucketPair to use StringTokenizer.

Test updates:
- test_exclusive_coordinator_plan: Ozone+EC blocks are 768MB, which is
  larger than all tables in our test environment. Use tpch_parquet which
  we rely on having 3 files (by loading from snapshot in this case).
- test_new_file_shorter: receives an EOFException when seeking with EC
- test_local_read: erasure-coded-bytes-read is also tied to IMPALA-11697
- test_erasure_coding: Ozone doesn't report files as erasure-coded
  (HDDS-7603)

Testing:
- Passes core E2E and custom cluster tests with TARGET_FILESYSTEM=ozone
  and ERASURE_CODING=true.

Change-Id: I201e2e33ce94bbc1e81631a0a315884bcc8047d1
Reviewed-on: http://gerrit.cloudera.org:8080/19324
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Support for Ozone with Erasure Coding
> -------------------------------------
>
>                 Key: IMPALA-11476
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11476
>             Project: IMPALA
>          Issue Type: Improvement
>            Reporter: Michael Smith
>            Assignee: Michael Smith
>            Priority: Major
>              Labels: ozone
>
> Similar to HDFS Erasure Coding, Ozone has the ability to use erasure coding rather than replication to store data: https://ci-hadoop.apache.org/view/Hadoop%20Ozone/job/ozone-doc-master/lastSuccessfulBuild/artifact/hadoop-hdds/docs/public/feature/erasurecoding.html.
> We have a mode to enable Erasure Coding, we should make it work with Ozone.
> Erasure Coding is not yet in a public release, although it's expected in CDP 7.1.8. Doing this will depend on having releases available to test with.
> May need a few updates to mark things as erasure-coded so Impala schedules them correctly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org