You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Imran Rashid (JIRA)" <ji...@apache.org> on 2015/03/05 19:16:38 UTC

[jira] [Commented] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

    [ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349191#comment-14349191 ] 

Imran Rashid commented on SPARK-6190:
-------------------------------------

Hi [~rxin], I've attached a design doc here.  I outlined a few proposals for dealing w/ ManagedBuffer, I think option #1 (creating a LargeManagedBuffer and removing LengthFieldBasedFrameDecoder) is the most straightforward.  I'm going to begin prototyping the decoder for larger messages.  Would appreciate any feedback on the design.

> create LargeByteBuffer abstraction for eliminating 2GB limit on blocks
> ----------------------------------------------------------------------
>
>                 Key: SPARK-6190
>                 URL: https://issues.apache.org/jira/browse/SPARK-6190
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Imran Rashid
>            Assignee: Imran Rashid
>         Attachments: LargeByteBuffer.pdf
>
>
> A key component in eliminating the 2GB limit on blocks is creating a proper abstraction for storing more than 2GB.  Currently spark is limited by a reliance on nio ByteBuffer and netty ByteBuf, both of which are limited at 2GB.  This task will introduce the new abstraction and the relevant implementation and utilities, without effecting the existing implementation at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org