You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Arun (JIRA)" <ji...@apache.org> on 2017/11/20 08:24:00 UTC

[jira] [Created] (SPARK-22561) Dynamically update topics list for spark kafka consumer

Arun created SPARK-22561:
----------------------------

             Summary: Dynamically update topics list for spark kafka consumer
                 Key: SPARK-22561
                 URL: https://issues.apache.org/jira/browse/SPARK-22561
             Project: Spark
          Issue Type: New Feature
          Components: DStreams
    Affects Versions: 2.2.0, 2.1.1, 2.1.0
            Reporter: Arun


The Spark Streaming application should allow to add new topic after streaming context is intialized and DStream is started.  This is very useful feature specially when business is working multi geography or  multi business units. 

For example initially I have spark-kakfa consumer listening for topics: ["topic-1"."topic-2"] and after couple of days I have added new topics to kafka ["topic-3","topic-4"], now is there a way to update spark-kafka consumer topics list and ask spark-kafka consumer to consume data for updated list of topics without stopping sparkStreaming application or sparkStreaming context.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org