You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2014/01/10 03:46:50 UTC
[1/6] git commit: Bug fixes for updating the RDD block's memory and
disk usage information
Updated Branches:
refs/heads/master d86a85e9c -> a9d533333
Bug fixes for updating the RDD block's memory and disk usage information
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/0af4b4f3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/0af4b4f3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/0af4b4f3
Branch: refs/heads/master
Commit: 0af4b4f3e86791dc47673be832e77e51b8a8ebcc
Parents: 0bc57c5
Author: walker <walker@transwarp01.(none)>
Authored: Wed Dec 25 20:07:01 2013 +0800
Committer: walker <walker@transwarp01.(none)>
Committed: Wed Dec 25 20:07:01 2013 +0800
----------------------------------------------------------------------
.../scala/org/apache/spark/storage/BlockManagerMasterActor.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/0af4b4f3/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
index 21022e1..1e44b1a 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
@@ -350,14 +350,15 @@ object BlockManagerMasterActor {
if (storageLevel.isValid) {
// isValid means it is either stored in-memory or on-disk.
- _blocks.put(blockId, BlockStatus(storageLevel, memSize, diskSize))
if (storageLevel.useMemory) {
+ _blocks.put(blockId, BlockStatus(storageLevel, memSize, 0))
_remainingMem -= memSize
logInfo("Added %s in memory on %s (size: %s, free: %s)".format(
blockId, blockManagerId.hostPort, Utils.bytesToString(memSize),
Utils.bytesToString(_remainingMem)))
}
if (storageLevel.useDisk) {
+ _blocks.put(blockId, BlockStatus(storageLevel, 0, diskSize))
logInfo("Added %s on disk on %s (size: %s)".format(
blockId, blockManagerId.hostPort, Utils.bytesToString(diskSize)))
}
[6/6] git commit: Merge pull request #294 from RongGu/master
Posted by rx...@apache.org.
Merge pull request #294 from RongGu/master
Bug fixes for updating the RDD block's memory and disk usage information
Bug fixes for updating the RDD block's memory and disk usage information.
>From the code context, we can find that the memSize and diskSize here are both always equal to the size of the block. Actually, they never be zero. Thus, the logic here is wrong for recording the block usage in BlockStatus, especially for the blocks which are dropped from memory to ensure space for the new input rdd blocks. I have tested it that this would cause the storage metrics shown in the Storage webpage wrong and misleading. With this patch, the metrics will be okay.
Finally, Merry Christmas, guys:)
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/a9d53333
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/a9d53333
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/a9d53333
Branch: refs/heads/master
Commit: a9d533333da63160c5b7507705b1b6e9c3dbf71e
Parents: d86a85e d942f95
Author: Reynold Xin <rx...@apache.org>
Authored: Thu Jan 9 18:46:46 2014 -0800
Committer: Reynold Xin <rx...@apache.org>
Committed: Thu Jan 9 18:46:46 2014 -0800
----------------------------------------------------------------------
.../org/apache/spark/storage/BlockManagerMasterActor.scala | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
[3/6] git commit: add inline comments
Posted by rx...@apache.org.
add inline comments
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/6ab1db80
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/6ab1db80
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/6ab1db80
Branch: refs/heads/master
Commit: 6ab1db8071101a097a3508861aaa12550a5bf9d5
Parents: a0c6d96
Author: walker <walker@transwarp01.(none)>
Authored: Tue Jan 7 01:21:25 2014 +0800
Committer: walker <walker@transwarp01.(none)>
Committed: Tue Jan 7 01:21:25 2014 +0800
----------------------------------------------------------------------
.../scala/org/apache/spark/storage/BlockManagerMasterActor.scala | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/6ab1db80/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
index 52a424d..a028783 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
@@ -348,6 +348,10 @@ object BlockManagerMasterActor {
if (storageLevel.isValid) {
// isValid means it is either stored in-memory or on-disk.
+ // But the memSize here indicates the data size in or dropped from memory,
+ // and the diskSize here indicates the data size in or dropped to disk.
+ // They can be both large than 0, when a block is dropped from memory to disk.
+ // Therefore, a safe way to set BlockStatus is to set its info in accurate modes.
if (storageLevel.useMemory) {
_blocks.put(blockId, BlockStatus(storageLevel, memSize, 0))
_remainingMem -= memSize
[5/6] git commit: Merge remote branch 'upstream/master'
Posted by rx...@apache.org.
Merge remote branch 'upstream/master'
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/d942f95d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/d942f95d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/d942f95d
Branch: refs/heads/master
Commit: d942f95d7ee39879ac326abf61b730f023506e4e
Parents: 2ad315e bb6a39a
Author: walker <walker@transwarp01.(none)>
Authored: Thu Jan 9 01:22:26 2014 +0800
Committer: walker <walker@transwarp01.(none)>
Committed: Thu Jan 9 01:22:26 2014 +0800
----------------------------------------------------------------------
.gitignore | 1 +
README.md | 20 +-
assembly/pom.xml | 6 +-
bin/pyspark | 2 +-
bin/run-example | 9 +-
bin/spark-class | 2 +-
conf/spark-env.sh.template | 2 +-
.../apache/spark/network/netty/FileClient.java | 5 +-
.../netty/FileClientChannelInitializer.java | 2 +-
.../apache/spark/network/netty/FileServer.java | 8 +-
.../netty/FileServerChannelInitializer.java | 4 +-
.../spark/network/netty/FileServerHandler.java | 6 +-
.../spark/network/netty/PathResolver.java | 52 +--
.../org/apache/spark/default-log4j.properties | 8 -
.../org/apache/spark/log4j-defaults.properties | 8 +
.../main/scala/org/apache/spark/Logging.scala | 10 +-
.../org/apache/spark/MapOutputTracker.scala | 14 +-
.../main/scala/org/apache/spark/SparkConf.scala | 13 +-
.../scala/org/apache/spark/SparkContext.scala | 4 +
.../main/scala/org/apache/spark/SparkEnv.scala | 16 +-
.../spark/deploy/ApplicationDescription.scala | 2 +-
.../apache/spark/deploy/client/TestClient.scala | 2 +-
.../spark/deploy/master/ApplicationInfo.scala | 7 +-
.../org/apache/spark/deploy/master/Master.scala | 11 +-
.../cluster/SparkDeploySchedulerBackend.scala | 2 +-
.../spark/storage/BlockManagerMaster.scala | 8 +-
.../apache/spark/storage/ThreadingTest.scala | 2 +-
.../scala/org/apache/spark/util/AkkaUtils.scala | 13 +-
.../apache/spark/MapOutputTrackerSuite.scala | 17 +-
.../apache/spark/deploy/JsonProtocolSuite.scala | 5 +-
.../deploy/worker/ExecutorRunnerTest.scala | 4 +-
.../spark/storage/BlockManagerSuite.scala | 12 +-
.../spark/storage/DiskBlockManagerSuite.scala | 4 +-
docs/README.md | 4 +-
docs/_plugins/copy_api_dirs.rb | 4 +-
docs/api.md | 2 +-
docs/configuration.md | 47 ++-
docs/css/bootstrap.min.css | 2 +-
docs/hadoop-third-party-distributions.md | 2 +-
docs/index.md | 6 +-
docs/job-scheduling.md | 5 +-
docs/mllib-guide.md | 330 +++++++++++++++----
docs/python-programming-guide.md | 2 +-
docs/quick-start.md | 8 +-
docs/running-on-yarn.md | 6 +-
docs/scala-programming-guide.md | 2 +-
docs/spark-standalone.md | 10 +
examples/pom.xml | 45 ++-
.../org/apache/spark/examples/JavaHdfsLR.java | 29 +-
.../org/apache/spark/examples/JavaKMeans.java | 23 +-
.../org/apache/spark/examples/JavaLogQuery.java | 20 +-
.../org/apache/spark/examples/JavaPageRank.java | 12 +-
.../org/apache/spark/examples/JavaSparkPi.java | 10 +-
.../java/org/apache/spark/examples/JavaTC.java | 16 +-
.../apache/spark/examples/JavaWordCount.java | 10 +-
.../apache/spark/mllib/examples/JavaALS.java | 18 +-
.../apache/spark/mllib/examples/JavaKMeans.java | 16 +-
.../org/apache/spark/mllib/examples/JavaLR.java | 19 +-
.../streaming/examples/JavaFlumeEventCount.java | 15 +-
.../streaming/examples/JavaKafkaWordCount.java | 23 +-
.../examples/JavaNetworkWordCount.java | 15 +-
.../streaming/examples/JavaQueueStream.java | 11 +-
.../streaming/examples/ActorWordCount.scala | 4 +-
.../streaming/examples/FlumeEventCount.scala | 3 +-
.../streaming/examples/HdfsWordCount.scala | 2 +-
.../streaming/examples/KafkaWordCount.scala | 5 +-
.../streaming/examples/MQTTWordCount.scala | 12 +-
.../streaming/examples/NetworkWordCount.scala | 2 +-
.../examples/StatefulNetworkWordCount.scala | 2 +-
.../streaming/examples/TwitterAlgebirdCMS.scala | 6 +-
.../streaming/examples/TwitterAlgebirdHLL.scala | 4 +-
.../streaming/examples/TwitterPopularTags.scala | 3 +-
.../streaming/examples/ZeroMQWordCount.scala | 13 +-
.../clickstream/PageViewGenerator.scala | 4 +-
.../examples/clickstream/PageViewStream.scala | 4 +-
external/flume/pom.xml | 93 ++++++
.../streaming/flume/FlumeInputDStream.scala | 155 +++++++++
.../spark/streaming/flume/FlumeUtils.scala | 70 ++++
.../streaming/flume/JavaFlumeStreamSuite.java | 34 ++
.../flume/src/test/resources/log4j.properties | 29 ++
.../streaming/flume/FlumeStreamSuite.scala | 86 +++++
external/kafka/pom.xml | 97 ++++++
.../streaming/kafka/KafkaInputDStream.scala | 154 +++++++++
.../spark/streaming/kafka/KafkaUtils.scala | 153 +++++++++
.../streaming/kafka/JavaKafkaStreamSuite.java | 45 +++
.../kafka/src/test/resources/log4j.properties | 29 ++
.../streaming/kafka/KafkaStreamSuite.scala | 39 +++
external/mqtt/pom.xml | 108 ++++++
.../spark/streaming/mqtt/MQTTInputDStream.scala | 110 +++++++
.../apache/spark/streaming/mqtt/MQTTUtils.scala | 75 +++++
.../streaming/mqtt/JavaMQTTStreamSuite.java | 37 +++
.../mqtt/src/test/resources/log4j.properties | 29 ++
.../spark/streaming/mqtt/MQTTStreamSuite.scala | 36 ++
external/twitter/pom.xml | 89 +++++
.../streaming/twitter/TwitterInputDStream.scala | 100 ++++++
.../spark/streaming/twitter/TwitterUtils.scala | 126 +++++++
.../twitter/JavaTwitterStreamSuite.java | 46 +++
.../twitter/src/test/resources/log4j.properties | 29 ++
.../streaming/twitter/TwitterStreamSuite.scala | 43 +++
external/zeromq/pom.xml | 89 +++++
.../spark/streaming/zeromq/ZeroMQReceiver.scala | 54 +++
.../spark/streaming/zeromq/ZeroMQUtils.scala | 126 +++++++
.../streaming/zeromq/JavaZeroMQStreamSuite.java | 50 +++
.../zeromq/src/test/resources/log4j.properties | 29 ++
.../streaming/zeromq/ZeroMQStreamSuite.scala | 44 +++
make-distribution.sh | 9 +-
.../spark/mllib/api/python/PythonMLLibAPI.scala | 30 ++
.../spark/mllib/optimization/Gradient.scala | 4 +-
.../MatrixFactorizationModel.scala | 38 ++-
.../optimization/GradientDescentSuite.scala | 116 +++++++
.../spark/mllib/recommendation/ALSSuite.scala | 33 +-
pom.xml | 32 +-
project/SparkBuild.scala | 103 ++++--
project/build.properties | 1 -
python/pyspark/mllib/_common.py | 25 ++
python/pyspark/mllib/recommendation.py | 12 +-
.../apache/spark/repl/SparkCommandLine.scala | 37 +++
.../org/apache/spark/repl/SparkILoop.scala | 7 +-
.../apache/spark/repl/SparkRunnerSettings.scala | 32 ++
sbin/stop-slaves.sh | 4 +-
sbt/sbt | 61 ++++
streaming/pom.xml | 83 ++---
.../spark/streaming/StreamingContext.scala | 145 +-------
.../api/java/JavaStreamingContext.scala | 243 +-------------
.../streaming/dstream/FlumeInputDStream.scala | 154 ---------
.../streaming/dstream/KafkaInputDStream.scala | 153 ---------
.../streaming/dstream/MQTTInputDStream.scala | 110 -------
.../streaming/dstream/TwitterInputDStream.scala | 99 ------
.../streaming/receivers/ZeroMQReceiver.scala | 53 ---
.../scheduler/NetworkInputTracker.scala | 4 +-
.../apache/spark/streaming/JavaAPISuite.java | 104 +-----
.../streaming/LocalJavaStreamingContext.java | 46 +++
.../spark/streaming/InputStreamsSuite.scala | 82 +----
.../apache/spark/streaming/TestSuiteBase.scala | 20 +-
134 files changed, 3492 insertions(+), 1609 deletions(-)
----------------------------------------------------------------------
[2/6] git commit: Merge remote branch 'upstream/master'
Posted by rx...@apache.org.
Merge remote branch 'upstream/master'
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/a0c6d96e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/a0c6d96e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/a0c6d96e
Branch: refs/heads/master
Commit: a0c6d96e270fc0bd26bfb7da0e3b2e40d934e1eb
Parents: 0af4b4f a2e7e04
Author: walker <walker@transwarp01.(none)>
Authored: Tue Jan 7 01:05:18 2014 +0800
Committer: walker <walker@transwarp01.(none)>
Committed: Tue Jan 7 01:05:18 2014 +0800
----------------------------------------------------------------------
.gitignore | 2 +
README.md | 28 +-
assembly/lib/PY4J_LICENSE.txt | 27 -
assembly/lib/PY4J_VERSION.txt | 1 -
assembly/lib/net/sf/py4j/py4j/0.7/py4j-0.7.jar | Bin 103286 -> 0 bytes
assembly/lib/net/sf/py4j/py4j/0.7/py4j-0.7.pom | 9 -
.../net/sf/py4j/py4j/maven-metadata-local.xml | 12 -
assembly/pom.xml | 14 +-
assembly/src/main/assembly/assembly.xml | 11 +-
bin/compute-classpath.cmd | 2 +-
bin/compute-classpath.sh | 2 +-
bin/pyspark | 70 ++
bin/pyspark.cmd | 23 +
bin/pyspark2.cmd | 55 +
bin/run-example | 91 ++
bin/run-example.cmd | 23 +
bin/run-example2.cmd | 61 ++
bin/slaves.sh | 91 --
bin/spark-class | 154 +++
bin/spark-class.cmd | 23 +
bin/spark-class2.cmd | 85 ++
bin/spark-config.sh | 36 -
bin/spark-daemon.sh | 183 ----
bin/spark-daemons.sh | 35 -
bin/spark-shell | 102 ++
bin/spark-shell.cmd | 23 +
bin/start-all.sh | 34 -
bin/start-master.sh | 52 -
bin/start-slave.sh | 35 -
bin/start-slaves.sh | 48 -
bin/stop-all.sh | 32 -
bin/stop-master.sh | 27 -
bin/stop-slaves.sh | 35 -
core/pom.xml | 422 ++++----
.../apache/spark/network/netty/FileClient.java | 32 +-
.../netty/FileClientChannelInitializer.java | 6 +-
.../spark/network/netty/FileClientHandler.java | 12 +-
.../apache/spark/network/netty/FileServer.java | 29 +-
.../netty/FileServerChannelInitializer.java | 3 +-
.../spark/network/netty/FileServerHandler.java | 18 +-
.../org/apache/spark/default-log4j.properties | 8 +
.../scala/org/apache/spark/Accumulators.scala | 8 +-
.../scala/org/apache/spark/HttpServer.scala | 1 +
.../main/scala/org/apache/spark/Logging.scala | 41 +-
.../org/apache/spark/MapOutputTracker.scala | 11 +-
.../scala/org/apache/spark/Partitioner.scala | 4 +-
.../main/scala/org/apache/spark/SparkConf.scala | 189 ++++
.../scala/org/apache/spark/SparkContext.scala | 338 +++---
.../main/scala/org/apache/spark/SparkEnv.scala | 54 +-
.../scala/org/apache/spark/TaskEndReason.scala | 2 -
.../org/apache/spark/api/java/JavaPairRDD.scala | 36 +
.../org/apache/spark/api/java/JavaRDDLike.scala | 11 +
.../spark/api/java/JavaSparkContext.scala | 56 +-
.../org/apache/spark/api/python/PythonRDD.scala | 4 +-
.../org/apache/spark/broadcast/Broadcast.scala | 8 +-
.../spark/broadcast/BroadcastFactory.scala | 4 +-
.../apache/spark/broadcast/HttpBroadcast.scala | 43 +-
.../spark/broadcast/TorrentBroadcast.scala | 45 +-
.../spark/deploy/FaultToleranceTest.scala | 4 +-
.../apache/spark/deploy/LocalSparkCluster.scala | 7 +-
.../apache/spark/deploy/SparkHadoopUtil.scala | 14 +-
.../org/apache/spark/deploy/client/Client.scala | 21 +-
.../apache/spark/deploy/client/TestClient.scala | 10 +-
.../org/apache/spark/deploy/master/Master.scala | 41 +-
.../spark/deploy/master/MasterArguments.scala | 11 +-
.../deploy/master/SparkZooKeeperSession.scala | 7 +-
.../master/ZooKeeperLeaderElectionAgent.scala | 9 +-
.../master/ZooKeeperPersistenceEngine.scala | 8 +-
.../spark/deploy/master/ui/MasterWebUI.scala | 2 +-
.../org/apache/spark/deploy/worker/Worker.scala | 34 +-
.../spark/deploy/worker/ui/WorkerWebUI.scala | 12 +-
.../executor/CoarseGrainedExecutorBackend.scala | 6 +-
.../org/apache/spark/executor/Executor.scala | 47 +-
.../org/apache/spark/io/CompressionCodec.scala | 19 +-
.../apache/spark/metrics/MetricsConfig.scala | 1 -
.../apache/spark/metrics/MetricsSystem.scala | 11 +-
.../spark/network/ConnectionManager.scala | 22 +-
.../org/apache/spark/network/ReceiverTest.scala | 12 +-
.../org/apache/spark/network/SenderTest.scala | 16 +-
.../spark/network/netty/ShuffleCopier.scala | 10 +-
.../org/apache/spark/rdd/CheckpointRDD.scala | 37 +-
.../org/apache/spark/rdd/CoGroupedRDD.scala | 2 +-
.../org/apache/spark/rdd/PairRDDFunctions.scala | 42 +
.../spark/rdd/PartitionerAwareUnionRDD.scala | 110 ++
.../main/scala/org/apache/spark/rdd/RDD.scala | 19 +-
.../apache/spark/rdd/RDDCheckpointData.scala | 17 +-
.../org/apache/spark/rdd/ShuffledRDD.scala | 2 +-
.../org/apache/spark/rdd/SubtractedRDD.scala | 2 +-
.../apache/spark/scheduler/DAGScheduler.scala | 47 +-
.../spark/scheduler/ExecutorLossReason.scala | 38 +
.../spark/scheduler/InputFormatInfo.scala | 14 +-
.../org/apache/spark/scheduler/JobLogger.scala | 6 +-
.../scala/org/apache/spark/scheduler/Pool.scala | 4 -
.../org/apache/spark/scheduler/ResultTask.scala | 4 +-
.../apache/spark/scheduler/Schedulable.scala | 1 -
.../spark/scheduler/SchedulableBuilder.scala | 6 +-
.../spark/scheduler/SchedulerBackend.scala | 34 +
.../apache/spark/scheduler/ShuffleMapTask.scala | 6 +-
.../apache/spark/scheduler/SparkListener.scala | 27 +-
.../spark/scheduler/SparkListenerBus.scala | 3 +-
.../spark/scheduler/TaskResultGetter.scala | 108 ++
.../apache/spark/scheduler/TaskScheduler.scala | 11 +-
.../spark/scheduler/TaskSchedulerImpl.scala | 472 +++++++++
.../apache/spark/scheduler/TaskSetManager.scala | 691 +++++++++++-
.../apache/spark/scheduler/WorkerOffer.scala | 24 +
.../scheduler/cluster/ClusterScheduler.scala | 473 ---------
.../cluster/ClusterTaskSetManager.scala | 713 -------------
.../cluster/CoarseGrainedSchedulerBackend.scala | 26 +-
.../scheduler/cluster/ExecutorLossReason.scala | 38 -
.../scheduler/cluster/SchedulerBackend.scala | 37 -
.../cluster/SimrSchedulerBackend.scala | 8 +-
.../cluster/SparkDeploySchedulerBackend.scala | 14 +-
.../scheduler/cluster/TaskResultGetter.scala | 108 --
.../spark/scheduler/cluster/WorkerOffer.scala | 24 -
.../mesos/CoarseMesosSchedulerBackend.scala | 23 +-
.../cluster/mesos/MesosSchedulerBackend.scala | 19 +-
.../spark/scheduler/local/LocalBackend.scala | 109 ++
.../spark/scheduler/local/LocalScheduler.scala | 222 ----
.../scheduler/local/LocalTaskSetManager.scala | 191 ----
.../spark/serializer/JavaSerializer.scala | 3 +-
.../spark/serializer/KryoSerializer.scala | 14 +-
.../apache/spark/serializer/Serializer.scala | 3 +
.../spark/serializer/SerializerManager.scala | 23 +-
.../spark/storage/BlockFetcherIterator.scala | 4 +-
.../org/apache/spark/storage/BlockManager.scala | 58 +-
.../spark/storage/BlockManagerMaster.scala | 11 +-
.../spark/storage/BlockManagerMasterActor.scala | 16 +-
.../spark/storage/BlockManagerWorker.scala | 3 -
.../spark/storage/BlockMessageArray.scala | 2 -
.../spark/storage/BlockObjectWriter.scala | 5 +-
.../apache/spark/storage/DiskBlockManager.scala | 2 +-
.../spark/storage/ShuffleBlockManager.scala | 35 +-
.../spark/storage/StoragePerfTester.scala | 2 +-
.../apache/spark/storage/ThreadingTest.scala | 8 +-
.../scala/org/apache/spark/ui/SparkUI.scala | 4 +-
.../apache/spark/ui/UIWorkloadGenerator.scala | 21 +-
.../org/apache/spark/ui/env/EnvironmentUI.scala | 15 +-
.../org/apache/spark/ui/exec/ExecutorsUI.scala | 31 +-
.../apache/spark/ui/jobs/ExecutorSummary.scala | 27 +
.../apache/spark/ui/jobs/ExecutorTable.scala | 90 ++
.../spark/ui/jobs/JobProgressListener.scala | 37 +-
.../org/apache/spark/ui/jobs/StagePage.scala | 5 +-
.../org/apache/spark/ui/jobs/StageTable.scala | 2 +-
.../scala/org/apache/spark/util/AkkaUtils.scala | 39 +-
.../org/apache/spark/util/MetadataCleaner.scala | 35 +-
.../spark/util/SerializableHyperLogLog.scala | 50 +
.../org/apache/spark/util/SizeEstimator.scala | 14 +-
.../apache/spark/util/TimeStampedHashMap.scala | 15 +-
.../scala/org/apache/spark/util/Utils.scala | 25 +-
core/src/test/resources/spark.conf | 8 +
.../test/resources/uncommons-maths-1.2.2.jar | Bin 49019 -> 0 bytes
.../org/apache/spark/CheckpointSuite.scala | 363 ++++---
.../scala/org/apache/spark/DriverSuite.scala | 8 +-
.../scala/org/apache/spark/FailureSuite.scala | 4 +-
.../org/apache/spark/FileServerSuite.scala | 108 +-
.../scala/org/apache/spark/JavaAPISuite.java | 36 +-
.../apache/spark/MapOutputTrackerSuite.scala | 16 +-
.../org/apache/spark/SharedSparkContext.scala | 4 +-
.../scala/org/apache/spark/SparkConfSuite.scala | 110 ++
.../SparkContextSchedulerCreationSuite.scala | 58 +-
.../deploy/worker/ExecutorRunnerTest.scala | 4 +-
.../apache/spark/io/CompressionCodecSuite.scala | 8 +-
.../spark/metrics/MetricsSystemSuite.scala | 8 +-
.../spark/rdd/PairRDDFunctionsSuite.scala | 34 +
.../scala/org/apache/spark/rdd/RDDSuite.scala | 40 +
.../spark/scheduler/ClusterSchedulerSuite.scala | 265 +++++
.../spark/scheduler/DAGSchedulerSuite.scala | 23 +-
.../org/apache/spark/scheduler/FakeTask.scala | 26 +
.../apache/spark/scheduler/JobLoggerSuite.scala | 4 +-
.../spark/scheduler/SparkListenerSuite.scala | 21 +-
.../spark/scheduler/TaskResultGetterSuite.scala | 112 ++
.../spark/scheduler/TaskSetManagerSuite.scala | 321 ++++++
.../cluster/ClusterSchedulerSuite.scala | 267 -----
.../cluster/ClusterTaskSetManagerSuite.scala | 319 ------
.../spark/scheduler/cluster/FakeTask.scala | 27 -
.../cluster/TaskResultGetterSuite.scala | 114 --
.../scheduler/local/LocalSchedulerSuite.scala | 227 ----
.../spark/serializer/KryoSerializerSuite.scala | 33 +-
.../spark/storage/BlockManagerSuite.scala | 97 +-
.../spark/storage/DiskBlockManagerSuite.scala | 18 +-
.../ui/jobs/JobProgressListenerSuite.scala | 73 ++
.../apache/spark/util/SizeEstimatorSuite.scala | 2 +-
data/kmeans_data.txt | 6 +
data/lr_data.txt | 1000 ++++++++++++++++++
data/pagerank_data.txt | 6 +
docs/README.md | 4 +-
docs/_config.yml | 2 +-
docs/_plugins/copy_api_dirs.rb | 4 +-
docs/api.md | 2 +-
docs/bagel-programming-guide.md | 4 +-
docs/building-with-maven.md | 14 +-
docs/configuration.md | 71 +-
docs/css/bootstrap.min.css | 2 +-
docs/hadoop-third-party-distributions.md | 2 +-
docs/index.md | 16 +-
docs/java-programming-guide.md | 4 +-
docs/job-scheduling.md | 21 +-
docs/mllib-guide.md | 2 +-
docs/monitoring.md | 3 +-
docs/python-programming-guide.md | 45 +-
docs/quick-start.md | 62 +-
docs/running-on-mesos.md | 19 +-
docs/running-on-yarn.md | 17 +-
docs/scala-programming-guide.md | 20 +-
docs/spark-debugger.md | 2 +-
docs/spark-standalone.md | 35 +-
docs/streaming-programming-guide.md | 8 +-
docs/tuning.md | 21 +-
ec2/spark_ec2.py | 2 +-
.../org/apache/spark/examples/JavaHdfsLR.java | 2 +-
.../org/apache/spark/examples/JavaKMeans.java | 2 +-
.../org/apache/spark/examples/JavaLogQuery.java | 2 +-
.../org/apache/spark/examples/JavaPageRank.java | 3 +-
.../org/apache/spark/examples/JavaSparkPi.java | 2 +-
.../java/org/apache/spark/examples/JavaTC.java | 2 +-
.../apache/spark/examples/JavaWordCount.java | 2 +-
.../apache/spark/mllib/examples/JavaALS.java | 2 +-
.../apache/spark/mllib/examples/JavaKMeans.java | 2 +-
.../org/apache/spark/mllib/examples/JavaLR.java | 2 +-
.../streaming/examples/JavaFlumeEventCount.java | 3 +-
.../streaming/examples/JavaKafkaWordCount.java | 7 +-
.../examples/JavaNetworkWordCount.java | 3 +-
.../streaming/examples/JavaQueueStream.java | 2 +-
.../apache/spark/examples/BroadcastTest.scala | 2 +-
.../spark/examples/ExceptionHandlingTest.scala | 2 +-
.../org/apache/spark/examples/GroupByTest.scala | 2 +-
.../org/apache/spark/examples/HBaseTest.scala | 2 +-
.../org/apache/spark/examples/HdfsTest.scala | 2 +-
.../org/apache/spark/examples/LogQuery.scala | 2 +-
.../spark/examples/MultiBroadcastTest.scala | 2 +-
.../examples/SimpleSkewedGroupByTest.scala | 2 +-
.../spark/examples/SkewedGroupByTest.scala | 2 +-
.../org/apache/spark/examples/SparkALS.scala | 2 +-
.../org/apache/spark/examples/SparkHdfsLR.scala | 2 +-
.../org/apache/spark/examples/SparkKMeans.scala | 2 +-
.../org/apache/spark/examples/SparkLR.scala | 2 +-
.../apache/spark/examples/SparkPageRank.scala | 2 +-
.../org/apache/spark/examples/SparkPi.scala | 2 +-
.../org/apache/spark/examples/SparkTC.scala | 2 +-
.../examples/bagel/WikipediaPageRank.scala | 10 +-
.../bagel/WikipediaPageRankStandalone.scala | 8 +-
.../streaming/examples/ActorWordCount.scala | 9 +-
.../streaming/examples/FlumeEventCount.scala | 2 +-
.../streaming/examples/HdfsWordCount.scala | 4 +-
.../streaming/examples/KafkaWordCount.scala | 4 +-
.../streaming/examples/MQTTWordCount.scala | 6 +-
.../streaming/examples/NetworkWordCount.scala | 4 +-
.../spark/streaming/examples/QueueStream.scala | 2 +-
.../streaming/examples/RawNetworkGrep.scala | 2 +-
.../examples/StatefulNetworkWordCount.scala | 4 +-
.../streaming/examples/TwitterAlgebirdCMS.scala | 2 +-
.../streaming/examples/TwitterAlgebirdHLL.scala | 2 +-
.../streaming/examples/TwitterPopularTags.scala | 2 +-
.../streaming/examples/ZeroMQWordCount.scala | 6 +-
.../clickstream/PageViewGenerator.scala | 4 +-
.../examples/clickstream/PageViewStream.scala | 6 +-
kmeans_data.txt | 6 -
lr_data.txt | 1000 ------------------
make-distribution.sh | 24 +-
.../spark/mllib/api/python/PythonMLLibAPI.scala | 232 ++++
.../spark/mllib/classification/NaiveBayes.scala | 119 +++
.../apache/spark/mllib/recommendation/ALS.scala | 13 +-
.../mllib/classification/NaiveBayesSuite.scala | 108 ++
new-yarn/pom.xml | 161 ---
.../spark/deploy/yarn/ApplicationMaster.scala | 446 --------
.../yarn/ApplicationMasterArguments.scala | 94 --
.../org/apache/spark/deploy/yarn/Client.scala | 519 ---------
.../spark/deploy/yarn/ClientArguments.scala | 149 ---
.../yarn/ClientDistributedCacheManager.scala | 228 ----
.../spark/deploy/yarn/WorkerLauncher.scala | 222 ----
.../spark/deploy/yarn/WorkerRunnable.scala | 209 ----
.../deploy/yarn/YarnAllocationHandler.scala | 687 ------------
.../spark/deploy/yarn/YarnSparkHadoopUtil.scala | 43 -
.../cluster/YarnClientClusterScheduler.scala | 47 -
.../cluster/YarnClientSchedulerBackend.scala | 109 --
.../cluster/YarnClusterScheduler.scala | 55 -
.../ClientDistributedCacheManagerSuite.scala | 220 ----
pagerank_data.txt | 6 -
pom.xml | 66 +-
project/SparkBuild.scala | 57 +-
pyspark | 70 --
pyspark.cmd | 23 -
pyspark2.cmd | 55 -
python/epydoc.conf | 2 +-
python/lib/py4j-0.8.1-src.zip | Bin 0 -> 37662 bytes
python/lib/py4j0.7.egg | Bin 191756 -> 0 bytes
python/pyspark/__init__.py | 34 +-
python/pyspark/broadcast.py | 11 +
python/pyspark/conf.py | 171 +++
python/pyspark/context.py | 68 +-
python/pyspark/java_gateway.py | 4 +-
python/pyspark/mllib/__init__.py | 20 +
python/pyspark/mllib/_common.py | 227 ++++
python/pyspark/mllib/classification.py | 86 ++
python/pyspark/mllib/clustering.py | 79 ++
python/pyspark/mllib/recommendation.py | 74 ++
python/pyspark/mllib/regression.py | 110 ++
python/pyspark/rdd.py | 66 +-
python/pyspark/serializers.py | 2 +-
python/pyspark/shell.py | 4 +-
python/pyspark/tests.py | 4 +-
python/run-tests | 3 +-
repl-bin/src/deb/bin/run | 3 +-
repl/pom.xml | 1 -
.../org/apache/spark/repl/SparkILoop.scala | 19 +-
.../org/apache/spark/repl/SparkIMain.scala | 7 +-
run-example | 91 --
run-example.cmd | 23 -
run-example2.cmd | 61 --
sbin/slaves.sh | 91 ++
sbin/spark-config.sh | 36 +
sbin/spark-daemon.sh | 183 ++++
sbin/spark-daemons.sh | 35 +
sbin/spark-executor | 23 +
sbin/start-all.sh | 34 +
sbin/start-master.sh | 52 +
sbin/start-slave.sh | 35 +
sbin/start-slaves.sh | 48 +
sbin/stop-all.sh | 32 +
sbin/stop-master.sh | 27 +
sbin/stop-slaves.sh | 35 +
sbt/sbt | 43 -
sbt/sbt-launch-0.11.3-2.jar | Bin 1096763 -> 0 bytes
sbt/sbt.cmd | 25 -
spark-class | 149 ---
spark-class.cmd | 23 -
spark-class2.cmd | 85 --
spark-executor | 22 -
spark-shell | 102 --
spark-shell.cmd | 22 -
.../org/apache/spark/streaming/Checkpoint.scala | 68 +-
.../org/apache/spark/streaming/DStream.scala | 15 +-
.../apache/spark/streaming/DStreamGraph.scala | 2 +-
.../scala/org/apache/spark/streaming/Job.scala | 41 -
.../org/apache/spark/streaming/JobManager.scala | 88 --
.../spark/streaming/NetworkInputTracker.scala | 174 ---
.../spark/streaming/PairDStreamFunctions.scala | 13 +-
.../org/apache/spark/streaming/Scheduler.scala | 131 ---
.../spark/streaming/StreamingContext.scala | 119 ++-
.../streaming/api/java/JavaPairDStream.scala | 18 +-
.../api/java/JavaStreamingContext.scala | 53 +-
.../streaming/dstream/FileInputDStream.scala | 153 +--
.../streaming/dstream/ForEachDStream.scala | 3 +-
.../streaming/dstream/NetworkInputDStream.scala | 9 +-
.../streaming/dstream/ShuffledDStream.scala | 9 +-
.../streaming/dstream/WindowedDStream.scala | 16 +-
.../spark/streaming/scheduler/BatchInfo.scala | 55 +
.../apache/spark/streaming/scheduler/Job.scala | 41 +
.../streaming/scheduler/JobGenerator.scala | 158 +++
.../streaming/scheduler/JobScheduler.scala | 106 ++
.../spark/streaming/scheduler/JobSet.scala | 68 ++
.../scheduler/NetworkInputTracker.scala | 175 +++
.../streaming/scheduler/StreamingListener.scala | 75 ++
.../scheduler/StreamingListenerBus.scala | 81 ++
.../streaming/util/MasterFailureTest.scala | 3 -
.../spark/streaming/util/RawTextSender.scala | 4 +-
.../apache/spark/streaming/JavaAPISuite.java | 10 +-
.../spark/streaming/BasicOperationsSuite.scala | 20 +-
.../spark/streaming/CheckpointSuite.scala | 85 +-
.../apache/spark/streaming/FailureSuite.scala | 13 +-
.../spark/streaming/InputStreamsSuite.scala | 32 +-
.../streaming/StreamingListenerSuite.scala | 71 ++
.../apache/spark/streaming/TestSuiteBase.scala | 48 +-
.../spark/streaming/WindowOperationsSuite.scala | 19 +-
yarn/README.md | 12 +
yarn/alpha/pom.xml | 32 +
.../spark/deploy/yarn/ApplicationMaster.scala | 464 ++++++++
.../org/apache/spark/deploy/yarn/Client.scala | 509 +++++++++
.../spark/deploy/yarn/WorkerLauncher.scala | 250 +++++
.../spark/deploy/yarn/WorkerRunnable.scala | 236 +++++
.../deploy/yarn/YarnAllocationHandler.scala | 680 ++++++++++++
.../yarn/ApplicationMasterArguments.scala | 94 ++
.../spark/deploy/yarn/ClientArguments.scala | 150 +++
.../yarn/ClientDistributedCacheManager.scala | 228 ++++
.../spark/deploy/yarn/YarnSparkHadoopUtil.scala | 43 +
.../cluster/YarnClientClusterScheduler.scala | 48 +
.../cluster/YarnClientSchedulerBackend.scala | 110 ++
.../cluster/YarnClusterScheduler.scala | 56 +
.../ClientDistributedCacheManagerSuite.scala | 220 ++++
yarn/pom.xml | 84 +-
.../spark/deploy/yarn/ApplicationMaster.scala | 477 ---------
.../yarn/ApplicationMasterArguments.scala | 94 --
.../org/apache/spark/deploy/yarn/Client.scala | 503 ---------
.../spark/deploy/yarn/ClientArguments.scala | 146 ---
.../yarn/ClientDistributedCacheManager.scala | 228 ----
.../spark/deploy/yarn/WorkerLauncher.scala | 243 -----
.../spark/deploy/yarn/WorkerRunnable.scala | 235 ----
.../deploy/yarn/YarnAllocationHandler.scala | 673 ------------
.../spark/deploy/yarn/YarnSparkHadoopUtil.scala | 43 -
.../cluster/YarnClientClusterScheduler.scala | 47 -
.../cluster/YarnClientSchedulerBackend.scala | 109 --
.../cluster/YarnClusterScheduler.scala | 55 -
.../ClientDistributedCacheManagerSuite.scala | 220 ----
yarn/stable/pom.xml | 32 +
.../spark/deploy/yarn/ApplicationMaster.scala | 432 ++++++++
.../org/apache/spark/deploy/yarn/Client.scala | 525 +++++++++
.../spark/deploy/yarn/WorkerLauncher.scala | 230 ++++
.../spark/deploy/yarn/WorkerRunnable.scala | 210 ++++
.../deploy/yarn/YarnAllocationHandler.scala | 695 ++++++++++++
399 files changed, 15450 insertions(+), 13942 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/a0c6d96e/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
----------------------------------------------------------------------
[4/6] git commit: add inline comments
Posted by rx...@apache.org.
add inline comments
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/2ad315e8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/2ad315e8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/2ad315e8
Branch: refs/heads/master
Commit: 2ad315e80fb297f1e6cf77ec6fcdcf25750b4a3d
Parents: 6ab1db8
Author: walker <walker@transwarp01.(none)>
Authored: Tue Jan 7 01:27:57 2014 +0800
Committer: walker <walker@transwarp01.(none)>
Committed: Tue Jan 7 01:27:57 2014 +0800
----------------------------------------------------------------------
.../scala/org/apache/spark/storage/BlockManagerMasterActor.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/2ad315e8/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
index a028783..2c1a4e2 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
@@ -350,7 +350,7 @@ object BlockManagerMasterActor {
// isValid means it is either stored in-memory or on-disk.
// But the memSize here indicates the data size in or dropped from memory,
// and the diskSize here indicates the data size in or dropped to disk.
- // They can be both large than 0, when a block is dropped from memory to disk.
+ // They can be both larger than 0, when a block is dropped from memory to disk.
// Therefore, a safe way to set BlockStatus is to set its info in accurate modes.
if (storageLevel.useMemory) {
_blocks.put(blockId, BlockStatus(storageLevel, memSize, 0))