[jira] [Updated] (CRUNCH-558) Add name to Spark Accumulators

["Micah Whitacre (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CRUNCH-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Micah Whitacre updated CRUNCH-558:
----------------------------------
    Attachment: CRUNCH-558.patch

Patch which sets the accumulator name to the pipeline name.

> Add name to Spark Accumulators
> ------------------------------
>
>                 Key: CRUNCH-558
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-558
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Spark
>            Reporter: Micah Whitacre
>            Assignee: Micah Whitacre
>         Attachments: CRUNCH-558.patch
>
>
> It was brought up on the mailing list that our Crunch counters are not showing up on the Spark webui possibly because they are not named.
> {quote}
> We are currently testing a few capabilities using Spark and one thing we noticed in Spark is they don't list any user defined accumulators on web UI. 
> On MapReduce I would imagine counters being displayed on the job page, however on a SparkPipeline I was only able to pull counter information from PipelineResult#getStageResult(). 
> I think the reason these accumulators are not visible on web UI is because crunch does not name these accumulators. Spark expects an accumulator to have a name to be visible on the UI.
> https://github.com/apache/crunch/blob/apache-crunch-0.13.0/crunch-spark/src/main/java/org/apache/crunch/impl/spark/SparkRuntime.java#L125-L126
> https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala#L616-L624 (accumulator API with Name)
> I would like to know if it's possible in crunch to name these accumulators so they are available in web UI. This will give us an experience where users can monitor/watch accumulators from web UI to obtain key information about their jobs. 
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)