You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/11 01:26:07 UTC
[50/50] git commit: Merge pull request #377 from andrewor14/master
Merge pull request #377 from andrewor14/master
External Sorting for Aggregator and CoGroupedRDDs (Revisited)
(This pull request is re-opened from https://github.com/apache/incubator-spark/pull/303, which was closed because Jenkins / github was misbehaving)
The target issue for this patch is the out-of-memory exceptions triggered by aggregate operations such as reduce, groupBy, join, and cogroup. The existing AppendOnlyMap used by these operations resides purely in memory, and grows with the size of the input data until the amount of allocated memory is exceeded. Under large workloads, this problem is aggravated by the fact that OOM frequently occurs only after a very long (> 1 hour) map phase, in which case the entire job must be restarted.
The solution is to spill the contents of this map to disk once a certain memory threshold is exceeded. This functionality is provided by ExternalAppendOnlyMap, which additionally sorts this buffer before writing it out to disk, and later merges these buffers back in sorted order.
Under normal circumstances in which OOM is not triggered, ExternalAppendOnlyMap is simply a wrapper around AppendOnlyMap and incurs little overhead. Only when the memory usage is expected to exceed the given threshold does ExternalAppendOnlyMap spill to disk.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/d37408f3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/d37408f3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/d37408f3
Branch: refs/heads/master
Commit: d37408f39ca3fd94f45b50a65f919f4d7007a533
Parents: 0eaf01c 2e393cd
Author: Patrick Wendell <pw...@gmail.com>
Authored: Fri Jan 10 16:25:01 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Fri Jan 10 16:25:01 2014 -0800
----------------------------------------------------------------------
.../scala/org/apache/spark/Aggregator.scala | 61 ++--
.../main/scala/org/apache/spark/SparkEnv.scala | 6 +-
.../org/apache/spark/executor/Executor.scala | 5 +
.../org/apache/spark/rdd/CoGroupedRDD.scala | 88 +++--
.../org/apache/spark/rdd/PairRDDFunctions.scala | 7 +-
.../org/apache/spark/storage/BlockId.scala | 12 +-
.../org/apache/spark/storage/BlockManager.scala | 2 +-
.../spark/storage/BlockObjectWriter.scala | 4 +
.../apache/spark/storage/DiskBlockManager.scala | 11 +-
.../org/apache/spark/util/AppendOnlyMap.scala | 237 -------------
.../spark/util/collection/AppendOnlyMap.scala | 297 ++++++++++++++++
.../util/collection/ExternalAppendOnlyMap.scala | 350 +++++++++++++++++++
.../collection/SizeTrackingAppendOnlyMap.scala | 101 ++++++
.../apache/spark/util/AppendOnlyMapSuite.scala | 154 --------
.../util/SizeTrackingAppendOnlyMapSuite.scala | 120 +++++++
.../util/collection/AppendOnlyMapSuite.scala | 198 +++++++++++
.../collection/ExternalAppendOnlyMapSuite.scala | 230 ++++++++++++
docs/configuration.md | 23 +-
.../apache/spark/streaming/JavaAPISuite.java | 29 +-
19 files changed, 1480 insertions(+), 455 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d37408f3/docs/configuration.md
----------------------------------------------------------------------