You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Zhan Zhang <zz...@hortonworks.com> on 2015/03/23 23:47:51 UTC

Review request for SPARK-6112:Provide OffHeap support through HDFS RAM_DISK

Hi Folks,

I am planning to implement hdfs off heap support for spark, and have uploaded the design doc for the off heap support through hdfs ramdisk in jira SPARK-6112. Please review it and provide your feedback if anybody are interested.

https://issues.apache.org/jira/browse/SPARK-6112

Thanks.

Zhan Zhang

Re: Review request for SPARK-6112:Provide OffHeap support through HDFS RAM_DISK

Posted by Zhan Zhang <zz...@hortonworks.com>.

Thanks Reynold,

Agree with you to open another JIRA to unify the block storage API.  I have upload the design doc to SPARK-6479 as well.

Thanks.

Zhan Zhang

On Mar 23, 2015, at 4:03 PM, Reynold Xin <rx...@databricks.com>> wrote:

I created a ticket to separate the API refactoring from the implementation. Would be great to have these as two separate patches to make it easier to review (similar to the way we are doing RPC refactoring -- first introducing an internal RPC api, port akka to it, and then add an alternative implementation).

https://issues.apache.org/jira/browse/SPARK-6479

Can you upload your design doc there so we can discuss the block store api? Thanks.


On Mon, Mar 23, 2015 at 3:47 PM, Zhan Zhang <zz...@hortonworks.com>> wrote:
Hi Folks,

I am planning to implement hdfs off heap support for spark, and have uploaded the design doc for the off heap support through hdfs ramdisk in jira SPARK-6112. Please review it and provide your feedback if anybody are interested.

https://issues.apache.org/jira/browse/SPARK-6112

Thanks.

Zhan Zhang

Re: Review request for SPARK-6112:Provide OffHeap support through HDFS RAM_DISK

Posted by Reynold Xin <rx...@databricks.com>.

I created a ticket to separate the API refactoring from the implementation.
Would be great to have these as two separate patches to make it easier to
review (similar to the way we are doing RPC refactoring -- first
introducing an internal RPC api, port akka to it, and then add an
alternative implementation).

https://issues.apache.org/jira/browse/SPARK-6479

Can you upload your design doc there so we can discuss the block store api?
Thanks.

On Mon, Mar 23, 2015 at 3:47 PM, Zhan Zhang <zz...@hortonworks.com> wrote:

> Hi Folks,
>
> I am planning to implement hdfs off heap support for spark, and have
> uploaded the design doc for the off heap support through hdfs ramdisk in
> jira SPARK-6112. Please review it and provide your feedback if anybody are
> interested.
>
> https://issues.apache.org/jira/browse/SPARK-6112
>
> Thanks.
>
> Zhan Zhang
>