You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Marcell Feher (JIRA)" <ji...@apache.org> on 2017/03/07 08:56:32 UTC

[jira] [Comment Edited] (HDFS-11503) Integrate Chocolate Cloud RS coder implementation

    [ https://issues.apache.org/jira/browse/HDFS-11503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898978#comment-15898978 ] 

Marcell Feher edited comment on HDFS-11503 at 3/7/17 8:56 AM:
--------------------------------------------------------------

We are working on fixing compatibility issues with the existing Reed-Solomon codecs, it is not an issue for this particular plugin. 

However I see a more general problem here. There are several other erasure coding algorithms like LRC whose integration should be possible by the HDFS-EC framework. (as a side-comment: our company specializes in using Random Linear Network Coding for storage and we are exploring the possibilities of an RLNC plugin to HDFS). 
My point: to make the system future-proof, I think the framework shouldn't assume that codecs are compatible with each other. Fundamentally different erasure coding algorithms should not be treated as an "exception" or "not RS" that needs to be guarded against. The ecosystem should be designed in a way where all codecs are treated as first class citizens, regardless of what algorithm they implement. It's a nice feature to tell the user which ones are byte-level compatible with each others, but this should not be a requirement. Who knows how awesome new EC algorithms will be developed in the future.


was (Author: sw0rdf1sh):
We are working on fixing compatibility issues with the existing Reed-Solomon codecs, it is not an issue for this particular plugin. 

However I see a more general problem here. There are several other erasure coding algorithms like LRC whose integration should be possible by the HDFS-EC framework. (as a side-comment: our company specializes in using Random Linear Network Coding for storage and we are exploring the possibilities of an RLNC plugin to HDFS). 
My point: to make the system future-proof, I think the framework shouldn't assume that codecs are compatible with each other. 

> Integrate Chocolate Cloud RS coder implementation
> -------------------------------------------------
>
>                 Key: HDFS-11503
>                 URL: https://issues.apache.org/jira/browse/HDFS-11503
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Andrew Wang
>            Assignee: Marcell Feher
>
> Quote from Marcell on HDFS-7285:
> First of all let me introduce ourselves: we are Chocolate Cloud from Denmark, we use erasure coding to improve storage solutions. We already have Reed-Solomon and Random Linear Network Coding backends for Liberasurecode, and now we are at the final stage of developing our RS plugin to HDFS-EC. The performance of our plugin is similar to ISA-L's, in some configurations we are better, in others we are worse (our initial speed comparison charts can be found here: https://www.chocolate-cloud.cc/Plugins/HDFS-EC/hdfs.html).
> We would like our plugin to become officially supported in Hadoop 3.0. We can already provide a preliminary version of our (native) library and a patch with the necessary glue code for the next alpha release.
> I'd like to know your thoughts about whether it's possible and how it could be achieved.
> P.S: I'm happy to share more details if there's interest



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org