You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrew Or (JIRA)" <ji...@apache.org> on 2015/05/19 08:02:00 UTC
[jira] [Created] (SPARK-7718) Speed up data source partitioning by
avoiding cleaning closures
Andrew Or created SPARK-7718:
--------------------------------
Summary: Speed up data source partitioning by avoiding cleaning closures
Key: SPARK-7718
URL: https://issues.apache.org/jira/browse/SPARK-7718
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.4.0
Reporter: Andrew Or
Assignee: Andrew Or
Priority: Critical
The new partitioning support strategy creates a bunch of RDDs (1 per partition, could be up to several thousands), then calls `mapPartitions` on every single one of these RDDs. This causes us to clean the same closure many times. Since we provide the closure in Spark we know for sure it is serializable, so we can bypass the cleaning for performance.
According to [~yhuai] cleaning 5000 closures take up to 6-7 seconds in a 12 seconds job that involves data source partitioning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org