You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2013/10/14 23:20:13 UTC
[6/6] git commit: Merge pull request #57 from aarondav/bid
Merge pull request #57 from aarondav/bid
Refactor BlockId into an actual type
Converts all of our BlockId strings into actual BlockId types. Here are some advantages of doing this now:
+ Type safety
+ Code clarity - it's now obvious what the key of a shuffle or rdd block is, for instance. Additionally, appearing in tuple/map type signatures is a big readability bonus. A Seq[(String, BlockStatus)] is not very clear. Further, we can now use more Scala features, like matching on BlockId types.
+ Explicit usage - we can now formally tell where various BlockIds are being used (without doing string searches); this makes updating current BlockIds a much clearer process, and compiler-supported.
(I'm looking at you, shuffle file consolidation.)
+ It will only get harder to make this change as time goes on.
Downside is, of course, that this is a very invasive change touching a lot of different files, which will inevitably lead to merge conflicts for many.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/3b11f43e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/3b11f43e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/3b11f43e
Branch: refs/heads/master
Commit: 3b11f43e36e2aca2346db7542c52fcbbeee70da2
Parents: 9979690 4a45019
Author: Reynold Xin <rx...@apache.org>
Authored: Mon Oct 14 14:20:01 2013 -0700
Committer: Reynold Xin <rx...@apache.org>
Committed: Mon Oct 14 14:20:01 2013 -0700
----------------------------------------------------------------------
.../spark/network/netty/FileClientHandler.java | 3 +-
.../spark/network/netty/FileServerHandler.java | 6 +-
.../apache/spark/BlockStoreShuffleFetcher.scala | 11 +-
.../scala/org/apache/spark/CacheManager.scala | 8 +-
.../spark/broadcast/BitTorrentBroadcast.scala | 4 +-
.../apache/spark/broadcast/HttpBroadcast.scala | 11 +-
.../apache/spark/broadcast/TreeBroadcast.scala | 8 +-
.../org/apache/spark/executor/Executor.scala | 4 +-
.../apache/spark/network/netty/FileHeader.scala | 22 ++--
.../spark/network/netty/ShuffleCopier.scala | 27 ++---
.../spark/network/netty/ShuffleSender.scala | 9 +-
.../scala/org/apache/spark/rdd/BlockRDD.scala | 6 +-
.../apache/spark/scheduler/DAGScheduler.scala | 6 +-
.../org/apache/spark/scheduler/TaskResult.scala | 3 +-
.../spark/serializer/KryoSerializer.scala | 10 +-
.../apache/spark/storage/BlockException.scala | 2 +-
.../spark/storage/BlockFetcherIterator.scala | 24 ++--
.../org/apache/spark/storage/BlockId.scala | 96 ++++++++++++++++
.../org/apache/spark/storage/BlockManager.scala | 95 +++++++---------
.../spark/storage/BlockManagerMaster.scala | 8 +-
.../spark/storage/BlockManagerMasterActor.scala | 21 ++--
.../spark/storage/BlockManagerMessages.scala | 16 +--
.../spark/storage/BlockManagerWorker.scala | 4 +-
.../org/apache/spark/storage/BlockMessage.scala | 38 +++----
.../spark/storage/BlockMessageArray.scala | 7 +-
.../spark/storage/BlockObjectWriter.scala | 2 +-
.../org/apache/spark/storage/BlockStore.scala | 14 +--
.../org/apache/spark/storage/DiskStore.scala | 37 +++---
.../org/apache/spark/storage/MemoryStore.scala | 34 +++---
.../spark/storage/ShuffleBlockManager.scala | 16 +--
.../org/apache/spark/storage/StorageUtils.scala | 47 ++++----
.../apache/spark/storage/ThreadingTest.scala | 6 +-
.../org/apache/spark/ui/storage/RDDPage.scala | 23 ++--
.../org/apache/spark/CacheManagerSuite.scala | 12 +-
.../org/apache/spark/CheckpointSuite.scala | 6 +-
.../org/apache/spark/DistributedSuite.scala | 16 +--
.../spark/scheduler/DAGSchedulerSuite.scala | 15 +--
.../cluster/TaskResultGetterSuite.scala | 3 +-
.../org/apache/spark/storage/BlockIdSuite.scala | 114 +++++++++++++++++++
.../spark/storage/BlockManagerSuite.scala | 102 +++++++++--------
.../spark/streaming/NetworkInputTracker.scala | 11 +-
.../streaming/dstream/NetworkInputDStream.scala | 14 +--
.../streaming/dstream/RawInputDStream.scala | 4 +-
.../streaming/receivers/ActorReceiver.scala | 4 +-
44 files changed, 544 insertions(+), 385 deletions(-)
----------------------------------------------------------------------