You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Anton Nekhaev (JIRA)" <ji...@apache.org> on 2015/08/10 17:44:45 UTC
[jira] [Updated] (SPARK-9781) KCL Workers should be configurable
from Spark configuration
[ https://issues.apache.org/jira/browse/SPARK-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anton Nekhaev updated SPARK-9781:
---------------------------------
Description:
Currently the KinesisClientLibConfiguration for KCL Workers is created within the KinesisReceiver and user is allowed to change only basic settings such as endpoint URL, stream name, credentials, etc.
However, there is no way to tune some advanced settings, e.g. MaxRecords, IdleTimeBetweenReads, FailoverTime, etc.
We can add these settings to the Spark configuration and parametrize KinesisClientLibConfiguration with them in KinesisReceiver.
was:
Currently the KinesisClientLibConfiguration for KCL Workers is created withing the KinesisReceiver and user is allowed to change only basic settings such as endpoint URL, stream name, credentials, etc.
However, there is no way to tune some advanced settings, e.g. MaxRecords, IdleTimeBetweenReads, FailoverTime, etc.
We can add this settings to the Spark configuration and parametrize KinesisClientLibConfiguration with them in KinesisReceiver.
> KCL Workers should be configurable from Spark configuration
> -----------------------------------------------------------
>
> Key: SPARK-9781
> URL: https://issues.apache.org/jira/browse/SPARK-9781
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.4.1
> Reporter: Anton Nekhaev
> Labels: kinesis
>
> Currently the KinesisClientLibConfiguration for KCL Workers is created within the KinesisReceiver and user is allowed to change only basic settings such as endpoint URL, stream name, credentials, etc.
> However, there is no way to tune some advanced settings, e.g. MaxRecords, IdleTimeBetweenReads, FailoverTime, etc.
> We can add these settings to the Spark configuration and parametrize KinesisClientLibConfiguration with them in KinesisReceiver.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org