[jira] [Comment Edited] (SPARK-2345) ForEachDStream should have an option of running the foreachfunc on Spark

["Tathagata Das (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14057824#comment-14057824 ] 

Tathagata Das edited comment on SPARK-2345 at 7/11/14 12:39 AM:
----------------------------------------------------------------

I must be missing something. If you want to run a Spark job inside the DStream.foreach function, you can just call any RDD action (count, collect, first, saveAsXXXXFile, etc.) inside that foreach function. context.runJob does not need to be called *explicitly*. All of the exisinting RDD actions (including the most generic rdd.foreachPartition) should be mostly sufficient for all requirements.

Maybe adding a code example of what you intend to do will help us disambiguate this?


was (Author: tdas):
I must be missing something. If you want to run a Spark job inside the DStream.foreach function, you can just call any RDD action (count, collect, first, saveAs***File, etc.) inside that foreach function. context.runJob does not need to be called *explicitly*. All of the exisinting RDD actions (including the most generic rdd.foreachPartition) should be mostly sufficient for all requirements.

Maybe adding a code example of what you intend to do will help us disambiguate this?

> ForEachDStream should have an option of running the foreachfunc on Spark
> ------------------------------------------------------------------------
>
>                 Key: SPARK-2345
>                 URL: https://issues.apache.org/jira/browse/SPARK-2345
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>            Reporter: Hari Shreedharan
>
> Today the Job generated simply calls the foreachfunc, but does not run it on spark itself using the sparkContext.runJob method.



--
This message was sent by Atlassian JIRA
(v6.2#6252)