[jira] [Created] (SPARK-20140) Remove hardcoded kinesis retry wait and max retries

["Yash Sharma (JIRA)" <ji...@apache.org>]

Yash Sharma created SPARK-20140:
-----------------------------------

             Summary: Remove hardcoded kinesis retry wait and max retries
                 Key: SPARK-20140
                 URL: https://issues.apache.org/jira/browse/SPARK-20140
             Project: Spark
          Issue Type: Bug
          Components: DStreams
    Affects Versions: 2.1.0
            Reporter: Yash Sharma


The pull requests proposes to remove the hardcoded values for Amazon Kinesis - MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES.

This change is critical for kinesis checkpoint recovery when the kinesis backed rdd is huge.
Following happens in a typical kinesis recovery :
- kinesis throttles large number of requests while recovering
- retries in case of throttling are not able to recover due to the small wait period
- kinesis throttles per second, the wait period should be configurable for recovery

The patch picks the spark kinesis configs from:
- spark.streaming.kinesis.retry.wait.time
- spark.streaming.kinesis.retry.max.attempts



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org