You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Tathagata Das (JIRA)" <ji...@apache.org> on 2015/09/12 02:51:45 UTC

[jira] [Resolved] (SPARK-7385) Add RDD.foreachPartitionWithIndex to allow the index to be used in transactions in streaming apps

     [ https://issues.apache.org/jira/browse/SPARK-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tathagata Das resolved SPARK-7385.
----------------------------------
    Resolution: Won't Fix

> Add RDD.foreachPartitionWithIndex to allow the index to be used in transactions in streaming apps
> -------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-7385
>                 URL: https://issues.apache.org/jira/browse/SPARK-7385
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, Streaming
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
>
> Spark Streaming apps often update external stores transactionally, which requires it to have an id that uniquely identifies the partition of data to be inserted. This can be the (time, partition index). 
> Current work around is to use mapPartitionsWithIndex().count() which is awkward as it is not an action. This jira is to add foreachPartitionWithIndex(). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org