You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Brian (JIRA)" <ji...@apache.org> on 2016/06/12 17:02:21 UTC
[jira] [Commented] (SPARK-6190) create LargeByteBuffer abstraction
for eliminating 2GB limit on blocks
[ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326532#comment-15326532 ]
Brian commented on SPARK-6190:
------------------------------
This is a terrible problem which I'm hoping someone will solve soon! We have jobs that take many hours to get to the point where they fail due to this bug. The only workaround we've found is to try to jiggle some of the spark settings like spark.sql.shuffle.partitions or manually repartition our dataframes/RDDs; however, since our datasets are not static we find that production jobs which have been running fine for months just suddenly die with this unexpected failure. Additionally, this bug makes it very difficult to have confidence that any of our new spark jobs will work - you basically have to try to run the job and if it fails start bumping up the partitions. This leads to a lot of stress when trying to meet project deadlines as jobs are failing because of a 2GB memory limit (it's really hard to explain to management that our jobs have memory problems even though our servers have 256 to 500 GB of ram each and we have close to 100 servers).
> create LargeByteBuffer abstraction for eliminating 2GB limit on blocks
> ----------------------------------------------------------------------
>
> Key: SPARK-6190
> URL: https://issues.apache.org/jira/browse/SPARK-6190
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Reporter: Imran Rashid
> Assignee: Imran Rashid
> Attachments: LargeByteBuffer_v3.pdf
>
>
> A key component in eliminating the 2GB limit on blocks is creating a proper abstraction for storing more than 2GB. Currently spark is limited by a reliance on nio ByteBuffer and netty ByteBuf, both of which are limited at 2GB. This task will introduce the new abstraction and the relevant implementation and utilities, without effecting the existing implementation at all.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org