You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@geode.apache.org by "Qihong Chen (JIRA)" <ji...@apache.org> on 2015/07/16 00:52:05 UTC

[jira] [Commented] (GEODE-120) RDD.saveToGemfire() can not handle big dataset (1M record per partition)

    [ https://issues.apache.org/jira/browse/GEODE-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628902#comment-14628902 ] 

Qihong Chen commented on GEODE-120:
-----------------------------------

post code review request: https://reviews.apache.org/r/36530/

> RDD.saveToGemfire() can not handle big dataset (1M record per partition)
> ------------------------------------------------------------------------
>
>                 Key: GEODE-120
>                 URL: https://issues.apache.org/jira/browse/GEODE-120
>             Project: Geode
>          Issue Type: Sub-task
>          Components: core, extensions
>    Affects Versions: 1.0.0-incubating
>            Reporter: Qihong Chen
>            Assignee: Qihong Chen
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> the connector use single region.putAll() call to save each RDD partition. But putAll() doesn't  handle big dataset well (such as 1M record). Need to split the dataset into smaller chunks, and invoke putAll() for each chunk. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)