[jira] [Resolved] (SPARK-8331) DirectStream wont work

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-8331.
------------------------------
    Resolution: Invalid

I think this should start as a question on user@ since there are really a bunch of questions here, not an issue report. There is no real info here about how you are running this or basic things like whether you're using more than 1 executor.

> DirectStream wont work
> ----------------------
>
>                 Key: SPARK-8331
>                 URL: https://issues.apache.org/jira/browse/SPARK-8331
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.4.0
>         Environment: all, Kafka 0.8.2.1
>            Reporter: sid
>
> I created a topic and pumped in messages using 
> kafka-console-producer < input file
> I use receiver based stream; and it works perfect
> JavaPairReceiverInputDStream<String, String> dstream = KafkaUtils.createStream(ssc, props.getProperty("zookeeper.connect"), props.getProperty("group.id"), topics);
> But when i use Direct Stream; it just sits there hanging
> JavaPairInputDStream<String, String> dstream = KafkaUtils.createDirectStream(ssc, String.class, String.class, StringDecoder.class, StringDecoder.class, params, topics);
>  
> my params like group.id are simply ignored
> Property auto.offset.reset is overridden to smallest
> Property client.id is overridden to sparkTestLocal
> Property group.id is overridden to 
> Property zookeeper.connect is overridden to
> Looking into code both KafkaCluster() using String encode/decode.
> Also since we are NOT passing number of partitions; i am guess all the 10 partitions i have should be receiving in 10 threads i created
> I even manually gen kafka-console-producer; while the DirectStream is running. 
> Just lies dead!!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org