[jira] [Commented] (HDFS-11503) Integrate Chocolate Cloud RS coder implementation

["Andrew Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-11503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897919#comment-15897919 ] 

Andrew Wang commented on HDFS-11503:
------------------------------------

From [~sw0rdf1sh]:

{quote}
Our understanding is that this is the intended way of shipping new native EC plugins. Some glue code is always necessary in the framework (like with Liberasurecode). Isn't this the case?
{quote}

Yes, for sure. Glue code is definitely fine. My concern though is that if CC's RS implementation isn't bit-identical to ISA-L and the Java coder, then users won't be able to transparently swap out the coder implementations. In this case, we'll need to add additional guardrails in the form of new "erasure coding policies" which encapsulate the coder and its parameters (e.g. CC-RS 6+3 with a 64KB cell size). We only have 5 supported policies right now, so adding CC-RS would greatly increase this configuration space.

> Integrate Chocolate Cloud RS coder implementation
> -------------------------------------------------
>
>                 Key: HDFS-11503
>                 URL: https://issues.apache.org/jira/browse/HDFS-11503
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Andrew Wang
>            Assignee: Marcell Feher
>
> Quote from Marcell on HDFS-7285:
> First of all let me introduce ourselves: we are Chocolate Cloud from Denmark, we use erasure coding to improve storage solutions. We already have Reed-Solomon and Random Linear Network Coding backends for Liberasurecode, and now we are at the final stage of developing our RS plugin to HDFS-EC. The performance of our plugin is similar to ISA-L's, in some configurations we are better, in others we are worse (our initial speed comparison charts can be found here: https://www.chocolate-cloud.cc/Plugins/HDFS-EC/hdfs.html).
> We would like our plugin to become officially supported in Hadoop 3.0. We can already provide a preliminary version of our (native) library and a patch with the necessary glue code for the next alpha release.
> I'd like to know your thoughts about whether it's possible and how it could be achieved.
> P.S: I'm happy to share more details if there's interest



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org