You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Brian London (JIRA)" <ji...@apache.org> on 2015/12/05 19:56:10 UTC
[jira] [Comment Edited] (SPARK-12134) Spark Streaming Kinesis:
Allow a mechanism to bypass validateRegion
[ https://issues.apache.org/jira/browse/SPARK-12134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15043445#comment-15043445 ]
Brian London edited comment on SPARK-12134 at 12/5/15 6:55 PM:
---------------------------------------------------------------
That's an interesting idea (although the {{return}} is unnecessary). I was thinking to convert it into a warning. I believe the region may be used in other ways later. {createStream} currently issues a warning if you specify both a region and a URL that the region extracted from the URL will overwrite the one specified -- something that should be fixed on its own. That logic will likely need to get changed as well. I'm not sure if it lives in Spark or in the Aws library. It's likely to be a problem if the later.
was (Author: brianlondon):
That's an interesting idea (although the {return} is unnecessary). I was thinking to convert it into a warning. I believe the region may be used in other ways later. {createStream} currently issues a warning if you specify both a region and a URL that the region extracted from the URL will overwrite the one specified -- something that should be fixed on its own. That logic will likely need to get changed as well. I'm not sure if it lives in Spark or in the Aws library. It's likely to be a problem if the later.
> Spark Streaming Kinesis: Allow a mechanism to bypass validateRegion
> -------------------------------------------------------------------
>
> Key: SPARK-12134
> URL: https://issues.apache.org/jira/browse/SPARK-12134
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.5.2
> Reporter: Brian London
>
> The {{KinesisUtils.createStream}} family of methods internally call a function {{validateRegion}} that requires the supplied region to be one of the strings on the list of AWS regions. Doing so, prevents the use of createStream to connect to a non-AWS kinesis stream such as one run through [Kinesalite|https://github.com/mhart/kinesalite], which can be useful for testing. The {{validateRegion}} method does no additional transformation so it's only effective function is to kill the createStream call if a user requests a stream with a non-standard region string (intentionally or otherwise).
> There should be a mechanism to bypass this check, if not remove it entirely.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org