[jira] [Created] (SPARK-11613) Kinesis ASL should allow caller to set ClientConfiguration for socket timeouts and other connection settings

["Heji Kim (JIRA)" <ji...@apache.org>]

Heji Kim created SPARK-11613:
--------------------------------

             Summary: Kinesis ASL should allow caller to set ClientConfiguration for socket timeouts and other connection settings
                 Key: SPARK-11613
                 URL: https://issues.apache.org/jira/browse/SPARK-11613
             Project: Spark
          Issue Type: Improvement
          Components: Streaming
    Affects Versions: 1.5.1
            Reporter: Heji Kim


Currently Kinesis ASL does not allow configuration of connection settings on AmazonKinesisClient such as socket timeout.

Currently we get intermittent 2 minute SocketTimeoutExceptions from Kinesis which can hold up the entire driver processing and pile up batch processing. We'd like to set it to something closer to the batch polling time of 5 seconds. The requests are normally subsecond so we'd rather fail fast.

KinesisBackedBlockRDD.KinesisSequenceRangeIterator uses only the credential based API to create AmazonKinesisClient and so we don't have access to com.amazonaws.ClientConfiguration which is the only API AWS provides for setting connection settings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org