[jira] [Commented] (SPARK-3639) Kinesis examples set master as local

["Chris Fregly (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193285#comment-14193285 ] 

Chris Fregly commented on SPARK-3639:
-------------------------------------

great catch, aniket!

for some background context, i was trying to make the sample easier to run out of the box.  i overlooked the spark-submit scenario, unfortunately.  thanks for fixing this.

few things:
1) does the Streaming Kinesis Guide (docs/streaming-kinesis-integration.md) need updating with your change?  specifically, the Running the Example section?  i don't think so, but something to double-check.

2) i noticed you put a comment in the scaladoc about needing +1 workers/threads than receivers, perhaps we should reword this to say 

  (number of kinesis shards+1) workers/threads are needed

because the number of receivers is determined by the number of shards in the kinesis stream.  might tighten up the message a bit.  

3)  should we throw an error if the number of workers/threads is not sufficient?  nobody likes an error message, but might be helpful here.  this is the basis of https://issues.apache.org/jira/browse/SPARK-2475, btw.  might want to keep an eye on that jira.

thanks again, man.  great catch.

-chris

> Kinesis examples set master as local
> ------------------------------------
>
>                 Key: SPARK-3639
>                 URL: https://issues.apache.org/jira/browse/SPARK-3639
>             Project: Spark
>          Issue Type: Bug
>          Components: Examples, Streaming
>    Affects Versions: 1.0.2, 1.1.0
>            Reporter: Aniket Bhatnagar
>            Assignee: Aniket Bhatnagar
>            Priority: Minor
>              Labels: examples
>             Fix For: 1.1.1, 1.2.0
>
>
> Kinesis examples set master as local thus not allowing the example to be tested on a cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org