You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by brucezhao11 <gi...@git.apache.org> on 2018/07/31 07:59:26 UTC
[GitHub] spark pull request #21929: [Streaming-24970][Kinesis] Create WriteAheadLogBa...
GitHub user brucezhao11 opened a pull request:
https://github.com/apache/spark/pull/21929
[Streaming-24970][Kinesis] Create WriteAheadLogBackedBlockRDD for Kinesis Streaming if WAL is enabled.
## What changes were proposed in this pull request?
By default, KinesisInputDStream creates KinesisBackedBlockRDD. When an application tries to recover from streaming checkpoint, the RDD will access Kinesis directly to re-read data. As all partitions in the BlockRDD accesses Kinesis, and AWS Kinesis only supports 5 concurrency reads per shard per second, it will touch ProvisionedThroughputExceededException easily. And when the conflicts are heavy, the recover will be failed.
Mostly, when we use Spark streaming, we will enable WAL. Then we can recover data from WAL, instead of re-reading from Kinesis directly.
This PR tries to create WriteAheadLogBackedBlockRDD for Kinesis Streaming if WAL is enabled.
## How was this patch tested?
In KinesisStreamSuite.scala, I will add a test case.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brucezhao11/spark dev
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21929.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21929
----
commit cbff70a803a59fce0d6121fabd0f5958d7b0436d
Author: bruce_zhao <br...@...>
Date: 2018-07-30T14:08:08Z
[Streaming][Kinesis] Create WriteAheadLogBackedBlockRDD for Kinesis Streaming if WAL is enabled. With this change, it can recover data from WAL, instead of re-reading from Kinesis directly when an application tries to recover from streaming checkpoint.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by brucezhao11 <gi...@git.apache.org>.
Github user brucezhao11 commented on the issue:
https://github.com/apache/spark/pull/21929
Thank you, @shuyang-truex . Really glad to hear that it works in your production environment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by brucezhao11 <gi...@git.apache.org>.
Github user brucezhao11 commented on the issue:
https://github.com/apache/spark/pull/21929
Hi, @brkyvz , @srowen , could you please review this PR?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by brucezhao11 <gi...@git.apache.org>.
Github user brucezhao11 commented on the issue:
https://github.com/apache/spark/pull/21929
@shengshuyang Yeah, just as you said, it shouldn't have any impacts after recovery, especially the life of receivers.
About the question, I mean receivers in one consumer. And any useful info in spark logs about the exit? Anyway, I believe you have already checked the details. Really strange.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21929
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by brucezhao11 <gi...@git.apache.org>.
Github user brucezhao11 commented on the issue:
https://github.com/apache/spark/pull/21929
@shengshuyang Just remember that we're running on Spark 2.2. I will check it on Spark 2.3 later. It shouldn't be a version problem, but worth a try.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21929
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by shengshuyang <gi...@git.apache.org>.
Github user shengshuyang commented on the issue:
https://github.com/apache/spark/pull/21929
@brucezhao11 Yup it was strange, after all your patch only touches the checkpoint recovery process, shouldn't cause any problem once it starts to process new batches. I did rebase your change onto 2.3.0 release though, wonder if that made a difference.
Each of our consumers only had one receiver process, and if you meant "if receivers across all consumers die at the same time", it's not the case either. Receivers die one at a time randomly. We have monitoring set up so our scheduling system kills and restarts a consumer when it happens, but checking our restart history, we cant' find any obvious pattern, except that the issue disappeared immediately after rollbacking to unpatched version.
Reading some documentation about spark internals, there should be a [receiver supervisor](https://jaceklaskowski.gitbooks.io/spark-streaming/content/spark-streaming-receiversupervisors.html) in charge of restarting receiver process, if you ran into the same issue in the future (hope not), it might worth digging into that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by shuyang-truex <gi...@git.apache.org>.
Github user shuyang-truex commented on the issue:
https://github.com/apache/spark/pull/21929
Thanks @brucezhao11. Unfortunately this patch did cause a side effect for us, where the receiver process would die randomly after running for a while (ranges from a few minutes to many hours, not during checkpoint recovery). Symptom being the streaming job still kept running, but with all empty batches and no API calls to Kinesis. We had to rollback and explore options like structured streaming in the end.
It does seem surprising that this important component doesn't work properly, but looks like the non-structured streaming is pretty much deprecated, the maintainers probably won't be very motivated to get it fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by shuyang-truex <gi...@git.apache.org>.
Github user shuyang-truex commented on the issue:
https://github.com/apache/spark/pull/21929
Just like to add that I was able to test this PR in our production environment and it works well, really hope to see it get merged.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21929
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [Streaming-24970][Kinesis] Create WriteAheadLogBackedBlo...
Posted by holdensmagicalunicorn <gi...@git.apache.org>.
Github user holdensmagicalunicorn commented on the issue:
https://github.com/apache/spark/pull/21929
@brucezhao11, thanks! I am a bot who has found some folks who might be able to help with the review:@tdas, @brkyvz and @huaxingao
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21929: [SPARK-24970][Kinesis] Create WriteAheadLogBackedBlockRD...
Posted by brucezhao11 <gi...@git.apache.org>.
Github user brucezhao11 commented on the issue:
https://github.com/apache/spark/pull/21929
Thanks for your update, @shuyang-truex . That's really strange. Did you mean all receivers died at the same time? So far, it works smoothly in our production for several months.
We ever got a similar symptom but was caused by another issue, which was related to a misusage about HDFS.
However, we will pay attention to the receivers' status. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org