You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tdas <gi...@git.apache.org> on 2015/04/28 05:35:55 UTC
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/5732
[SPARK-7139][Streaming] Allow received block metadata to be saved to WAL and recovered on driver failure
- Enabled ReceivedBlockTracker WAL by default
- Stored block metadata in the WAL
- Optimized WALBackedBlockRDD by skipping block fetch when the block is known to not exist in Spark
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-7139
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5732.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5732
----
commit d06fa21f80eeda04c11e454a461902717a2e5c7d
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-04-28T03:33:59Z
Enabled ReceivedBlockTracker by default, stored block metadata and optimized block fetching in WALBackedBlockRDD
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-96907713
[Test build #31106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31106/consoleFull) for PR 5732 at commit [`d06fa21`](https://github.com/apache/spark/commit/d06fa21f80eeda04c11e454a461902717a2e5c7d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97381605
[Test build #31266 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31266/consoleFull) for PR 5732 at commit [`5f67a59`](https://github.com/apache/spark/commit/5f67a59c3091c774ec3524677f53192a78fbbc20).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **adds the following new dependencies:**
* `spark-unsafe_2.10-1.4.0-SNAPSHOT.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-96898576
@harishreedharan @jerryshao
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-96968949
[Test build #31135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31135/consoleFull) for PR 5732 at commit [`1bc5bc3`](https://github.com/apache/spark/commit/1bc5bc384bc5baa9d904588b36dd08393bc3c91f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/5732#discussion_r29429755
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/ReceiverInputDStream.scala ---
@@ -67,27 +67,26 @@ abstract class ReceiverInputDStream[T: ClassTag](@transient ssc_ : StreamingCont
} else {
// Otherwise, ask the tracker for all the blocks that have been allocated to this stream
// for this batch
- val blockInfos =
- ssc.scheduler.receiverTracker.getBlocksOfBatch(validTime).get(id).getOrElse(Seq.empty)
- val blockStoreResults = blockInfos.map { _.blockStoreResult }
- val blockIds = blockStoreResults.map { _.blockId.asInstanceOf[BlockId] }.toArray
+ val receiverTracker = ssc.scheduler.receiverTracker
+ val blockInfos = receiverTracker.getBlocksOfBatch(validTime).getOrElse(id, Seq.empty)
+ val blockIds = blockInfos.map { _.blockId.asInstanceOf[BlockId] }.toArray
- // Check whether all the results are of the same type
- val resultTypes = blockStoreResults.map { _.getClass }.distinct
- if (resultTypes.size > 1) {
- logWarning("Multiple result types in block information, WAL information will be ignored.")
- }
+ // Is WAL segment info present with all the blocks
+ val isWALSegmentInfoPresent = blockInfos.forall { _.walRecordHandleOption.nonEmpty }
- // If all the results are of type WriteAheadLogBasedStoreResult, then create
- // WriteAheadLogBackedBlockRDD else create simple BlockRDD.
- if (resultTypes.size == 1 && resultTypes.head == classOf[WriteAheadLogBasedStoreResult]) {
- val logSegments = blockStoreResults.map {
- _.asInstanceOf[WriteAheadLogBasedStoreResult].walRecordHandle
- }.toArray
- // Since storeInBlockManager = false, the storage level does not matter.
- new WriteAheadLogBackedBlockRDD[T](ssc.sparkContext,
- blockIds, logSegments, storeInBlockManager = false, StorageLevel.MEMORY_ONLY_SER)
+ if (isWALSegmentInfoPresent) {
+ // If all the blocks have WAL segment info, then create a WALBackedBlockRDD
+ val isBlockIdValid = blockInfos.map { _.isBlockIdValid() }.toArray
+ val blockWALSegments = blockInfos.map { _.walRecordHandleOption.get }.toArray
--- End diff --
Also here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/5732#discussion_r29429719
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/ReceiverInputDStream.scala ---
@@ -67,27 +67,26 @@ abstract class ReceiverInputDStream[T: ClassTag](@transient ssc_ : StreamingCont
} else {
// Otherwise, ask the tracker for all the blocks that have been allocated to this stream
// for this batch
- val blockInfos =
- ssc.scheduler.receiverTracker.getBlocksOfBatch(validTime).get(id).getOrElse(Seq.empty)
- val blockStoreResults = blockInfos.map { _.blockStoreResult }
- val blockIds = blockStoreResults.map { _.blockId.asInstanceOf[BlockId] }.toArray
+ val receiverTracker = ssc.scheduler.receiverTracker
+ val blockInfos = receiverTracker.getBlocksOfBatch(validTime).getOrElse(id, Seq.empty)
+ val blockIds = blockInfos.map { _.blockId.asInstanceOf[BlockId] }.toArray
- // Check whether all the results are of the same type
- val resultTypes = blockStoreResults.map { _.getClass }.distinct
- if (resultTypes.size > 1) {
- logWarning("Multiple result types in block information, WAL information will be ignored.")
- }
+ // Is WAL segment info present with all the blocks
+ val isWALSegmentInfoPresent = blockInfos.forall { _.walRecordHandleOption.nonEmpty }
--- End diff --
is it better to rename as `isWALRecordHandleInfoPresent` ? Looked though this part of code, some variables use `segment` but actually is `handle`, might be better to keep consistent :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97381618
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31266/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97636108
[Test build #31360 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31360/consoleFull) for PR 5732 at commit [`466212c`](https://github.com/apache/spark/commit/466212cca8a017b19fe4821be46f2027de08833f).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97630027
@jerryshao @harishreedharan Please take a look at this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97621839
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97636118
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31360/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-96986841
[Test build #31135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31135/consoleFull) for PR 5732 at commit [`1bc5bc3`](https://github.com/apache/spark/commit/1bc5bc384bc5baa9d904588b36dd08393bc3c91f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5732#discussion_r29462598
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/ReceiverInputDStream.scala ---
@@ -67,27 +67,26 @@ abstract class ReceiverInputDStream[T: ClassTag](@transient ssc_ : StreamingCont
} else {
// Otherwise, ask the tracker for all the blocks that have been allocated to this stream
// for this batch
- val blockInfos =
- ssc.scheduler.receiverTracker.getBlocksOfBatch(validTime).get(id).getOrElse(Seq.empty)
- val blockStoreResults = blockInfos.map { _.blockStoreResult }
- val blockIds = blockStoreResults.map { _.blockId.asInstanceOf[BlockId] }.toArray
+ val receiverTracker = ssc.scheduler.receiverTracker
+ val blockInfos = receiverTracker.getBlocksOfBatch(validTime).getOrElse(id, Seq.empty)
+ val blockIds = blockInfos.map { _.blockId.asInstanceOf[BlockId] }.toArray
- // Check whether all the results are of the same type
- val resultTypes = blockStoreResults.map { _.getClass }.distinct
- if (resultTypes.size > 1) {
- logWarning("Multiple result types in block information, WAL information will be ignored.")
- }
+ // Is WAL segment info present with all the blocks
+ val isWALSegmentInfoPresent = blockInfos.forall { _.walRecordHandleOption.nonEmpty }
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97951292
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5732#discussion_r29398534
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/HdfsUtils.scala ---
@@ -27,7 +27,7 @@ private[streaming] object HdfsUtils {
// If the file exists and we have append support, append instead of creating a new file
val stream: FSDataOutputStream = {
if (dfs.isFile(dfsPath)) {
- if (conf.getBoolean("hdfs.append.support", false)) {
+ if (conf.getBoolean("hdfs.append.support", false) || dfs.isInstanceOf[RawLocalFileSystem]) {
--- End diff --
@harishreedharan This change is to enable append when running locally. Without this the unit tests fail because all the units are running on manual clock and so all of them (when they reuse the same checkpoint directory) try to write to log files with the same name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97951321
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97621859
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97978656
[Test build #31446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31446/consoleFull) for PR 5732 at commit [`637bc9c`](https://github.com/apache/spark/commit/637bc9c994cd1dbc74f433cb511ba131c85b4ee2).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-96899198
[Test build #31106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31106/consoleFull) for PR 5732 at commit [`d06fa21`](https://github.com/apache/spark/commit/d06fa21f80eeda04c11e454a461902717a2e5c7d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97351869
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97978708
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97978712
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31446/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97381616
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5732#discussion_r29462725
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala ---
@@ -40,21 +40,29 @@ private[streaming]
class WriteAheadLogBackedBlockRDDPartition(
val index: Int,
val blockId: BlockId,
- val walRecordHandle: WriteAheadLogRecordHandle)
- extends Partition
+ val isBlockIdValid: Boolean,
+ val walRecordHandle: WriteAheadLogRecordHandle
+ ) extends Partition
/**
* This class represents a special case of the BlockRDD where the data blocks in
* the block manager are also backed by data in write ahead logs. For reading
* the data, this RDD first looks up the blocks by their ids in the block manager.
- * If it does not find them, it looks up the corresponding data in the write ahead log.
+ * If it does not find them, it looks up the corresponding file segment. The finding
--- End diff --
Changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5732#discussion_r29462609
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/ReceiverInputDStream.scala ---
@@ -67,27 +67,26 @@ abstract class ReceiverInputDStream[T: ClassTag](@transient ssc_ : StreamingCont
} else {
// Otherwise, ask the tracker for all the blocks that have been allocated to this stream
// for this batch
- val blockInfos =
- ssc.scheduler.receiverTracker.getBlocksOfBatch(validTime).get(id).getOrElse(Seq.empty)
- val blockStoreResults = blockInfos.map { _.blockStoreResult }
- val blockIds = blockStoreResults.map { _.blockId.asInstanceOf[BlockId] }.toArray
+ val receiverTracker = ssc.scheduler.receiverTracker
+ val blockInfos = receiverTracker.getBlocksOfBatch(validTime).getOrElse(id, Seq.empty)
+ val blockIds = blockInfos.map { _.blockId.asInstanceOf[BlockId] }.toArray
- // Check whether all the results are of the same type
- val resultTypes = blockStoreResults.map { _.getClass }.distinct
- if (resultTypes.size > 1) {
- logWarning("Multiple result types in block information, WAL information will be ignored.")
- }
+ // Is WAL segment info present with all the blocks
+ val isWALSegmentInfoPresent = blockInfos.forall { _.walRecordHandleOption.nonEmpty }
- // If all the results are of type WriteAheadLogBasedStoreResult, then create
- // WriteAheadLogBackedBlockRDD else create simple BlockRDD.
- if (resultTypes.size == 1 && resultTypes.head == classOf[WriteAheadLogBasedStoreResult]) {
- val logSegments = blockStoreResults.map {
- _.asInstanceOf[WriteAheadLogBasedStoreResult].walRecordHandle
- }.toArray
- // Since storeInBlockManager = false, the storage level does not matter.
- new WriteAheadLogBackedBlockRDD[T](ssc.sparkContext,
- blockIds, logSegments, storeInBlockManager = false, StorageLevel.MEMORY_ONLY_SER)
+ if (isWALSegmentInfoPresent) {
+ // If all the blocks have WAL segment info, then create a WALBackedBlockRDD
+ val isBlockIdValid = blockInfos.map { _.isBlockIdValid() }.toArray
+ val blockWALSegments = blockInfos.map { _.walRecordHandleOption.get }.toArray
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97951596
[Test build #31446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31446/consoleFull) for PR 5732 at commit [`637bc9c`](https://github.com/apache/spark/commit/637bc9c994cd1dbc74f433cb511ba131c85b4ee2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97352082
[Test build #31266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31266/consoleFull) for PR 5732 at commit [`5f67a59`](https://github.com/apache/spark/commit/5f67a59c3091c774ec3524677f53192a78fbbc20).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97622053
[Test build #31360 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31360/consoleFull) for PR 5732 at commit [`466212c`](https://github.com/apache/spark/commit/466212cca8a017b19fe4821be46f2027de08833f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/5732#discussion_r29430355
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala ---
@@ -40,21 +40,29 @@ private[streaming]
class WriteAheadLogBackedBlockRDDPartition(
val index: Int,
val blockId: BlockId,
- val walRecordHandle: WriteAheadLogRecordHandle)
- extends Partition
+ val isBlockIdValid: Boolean,
+ val walRecordHandle: WriteAheadLogRecordHandle
+ ) extends Partition
/**
* This class represents a special case of the BlockRDD where the data blocks in
* the block manager are also backed by data in write ahead logs. For reading
* the data, this RDD first looks up the blocks by their ids in the block manager.
- * If it does not find them, it looks up the corresponding data in the write ahead log.
+ * If it does not find them, it looks up the corresponding file segment. The finding
--- End diff --
should we here specifically mention "file segment", or just `walRecordHandles`? Since WAL is pluggable, it might not have to be file segment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5732#discussion_r29398445
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala ---
@@ -94,47 +111,53 @@ class WriteAheadLogBackedBlockRDD[T: ClassTag](
val blockManager = SparkEnv.get.blockManager
val partition = split.asInstanceOf[WriteAheadLogBackedBlockRDDPartition]
val blockId = partition.blockId
- blockManager.get(blockId) match {
- case Some(block) => // Data is in Block Manager
- val iterator = block.data.asInstanceOf[Iterator[T]]
- logDebug(s"Read partition data of $this from block manager, block $blockId")
- iterator
- case None => // Data not found in Block Manager, grab it from write ahead log file
- var dataRead: ByteBuffer = null
- var writeAheadLog: WriteAheadLog = null
- try {
- // The WriteAheadLogUtils.createLog*** method needs a directory to create a
- // WriteAheadLog object as the default FileBasedWriteAheadLog needs a directory for
- // writing log data. However, the directory is not needed if data needs to be read, hence
- // a dummy path is provided to satisfy the method parameter requirements.
- // FileBasedWriteAheadLog will not create any file or directory at that path.
- val dummyDirectory = FileUtils.getTempDirectoryPath()
- writeAheadLog = WriteAheadLogUtils.createLogForReceiver(
- SparkEnv.get.conf, dummyDirectory, hadoopConf)
- dataRead = writeAheadLog.read(partition.walRecordHandle)
- } catch {
- case NonFatal(e) =>
- throw new SparkException(
- s"Could not read data from write ahead log record ${partition.walRecordHandle}", e)
- } finally {
- if (writeAheadLog != null) {
- writeAheadLog.close()
- writeAheadLog = null
- }
- }
- if (dataRead == null) {
+
+ def getBlockFromBlockManager(): Option[Iterator[T]] = {
--- End diff --
This whole thing just puts the old code within inner functions, not really a big change. The real change is that these functions are called selectively in lines 157..160.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97636116
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7139][Streaming] Allow received block m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5732#issuecomment-97351883
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org