You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "William Benton (JIRA)" <ji...@apache.org> on 2014/04/09 20:36:17 UTC
[jira] [Created] (SPARK-1454) PySpark accumulators fail to update
when runJob takes serialized/captured closures
William Benton created SPARK-1454:
-------------------------------------
Summary: PySpark accumulators fail to update when runJob takes serialized/captured closures
Key: SPARK-1454
URL: https://issues.apache.org/jira/browse/SPARK-1454
Project: Spark
Issue Type: Bug
Components: PySpark, Spark Core
Affects Versions: 1.0.0
Reporter: William Benton
Priority: Minor
My patch for SPARK-729 optionally serializes closures when they are cleaned (in order to capture the values of mutable free variables at declaration time rather than at execution time). This behavior is currently disabled for the closure argument to SparkContext.runJob, because enabling it there causes Python accumulators to fail to update.
The purpose of this JIRA is to note this issue and fix whatever is causing Python accumulators to behave this way so that closures passed to runJob can be captured in general.
--
This message was sent by Atlassian JIRA
(v6.2#6252)