[jira] [Commented] (SPARK-4368) Ceph integration?

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290644#comment-14290644 ] 

Sean Owen commented on SPARK-4368:
----------------------------------

I don't think Spark does anything in particular to support GlusterFS; the message you cite just says it works without any special support. I haven't heard Ceph come up. Are you suggesting there is some change that needs to be made to support it? if so I think you should outline how big the change is. I think the suggestion recently has been that third-party integration projects belong outside the core project though.

> Ceph integration?
> -----------------
>
>                 Key: SPARK-4368
>                 URL: https://issues.apache.org/jira/browse/SPARK-4368
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output
>            Reporter: Serge Smertin
>
> There is a use-case of storing big number of relatively small BLOB objects (2-20Mb), which has to have some ugly workarounds in HDFS environments. There is a need to process those BLOBs close to data themselves, so that's why MapReduce paradigm is good, as it guarantees data locality.
> Ceph seems to be one of the systems that maintains both of the properties (small files and data locality) -  http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-July/032119.html. I know already that Spark supports GlusterFS - http://mail-archives.apache.org/mod_mbox/spark-user/201404.mbox/%3CCF657F2B.5B3A1%25venkat@yarcdata.com%3E
> So i wonder, could there be an integration with this storage solution and what could be the effort of doing that? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org