You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ashok Kumar <as...@yahoo.com.INVALID> on 2017/06/19 19:00:36 UTC

how many topics spark streaming can handle

 Hi Gurus,
Within one Spark streaming process how many topics can be handled? I have not tried more than one topic.
Thanks

Re: how many topics spark streaming can handle

Posted by Bryan Jeffrey <br...@gmail.com>.
Hello Ashok, 




We're consuming from more than 10 topics in some Spark streaming applications. Topic management is a concern (what is read from where, etc), but I have seen no issues from Spark itself. 




Regards, 




Bryan Jeffrey 




Get Outlook for Android







On Mon, Jun 19, 2017 at 3:24 PM -0400, "Ashok Kumar" <as...@yahoo.com.invalid> wrote:










thank you
in the following example
   val topics = "test1,test2,test3"
    val brokers = "localhost:9092"
    val topicsSet = topics.split(",").toSet
    val sparkConf = new SparkConf().setAppName("KafkaDroneCalc").setMaster("local") //spark://localhost:7077
    val sc = new SparkContext(sparkConf)
    val ssc = new StreamingContext(sc, Seconds(30))
    val kafkaParams = Map[String, String]("metadata.broker.list" -> brokers)
    val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder] (ssc, kafkaParams, topicsSet)
  it is possible to have three topics or many topics?

 

    On Monday, 19 June 2017, 20:10, Michael Armbrust <mi...@databricks.com> wrote:
  

 I don't think that there is really a Spark specific limit here.  It would be a function of the size of your spark / kafka clusters and the type of processing you are trying to do.
On Mon, Jun 19, 2017 at 12:00 PM, Ashok Kumar <as...@yahoo.com.invalid> wrote:
  Hi Gurus,
Within one Spark streaming process how many topics can be handled? I have not tried more than one topic.
Thanks

Re: how many topics spark streaming can handle

Posted by Ashok Kumar <as...@yahoo.com.INVALID>.
thank you
in the following example
   val topics = "test1,test2,test3"
    val brokers = "localhost:9092"
    val topicsSet = topics.split(",").toSet
    val sparkConf = new SparkConf().setAppName("KafkaDroneCalc").setMaster("local") //spark://localhost:7077
    val sc = new SparkContext(sparkConf)
    val ssc = new StreamingContext(sc, Seconds(30))
    val kafkaParams = Map[String, String]("metadata.broker.list" -> brokers)
    val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder] (ssc, kafkaParams, topicsSet)
  it is possible to have three topics or many topics?

 

    On Monday, 19 June 2017, 20:10, Michael Armbrust <mi...@databricks.com> wrote:
 

 I don't think that there is really a Spark specific limit here.  It would be a function of the size of your spark / kafka clusters and the type of processing you are trying to do.
On Mon, Jun 19, 2017 at 12:00 PM, Ashok Kumar <as...@yahoo.com.invalid> wrote:

 Hi Gurus,
Within one Spark streaming process how many topics can be handled? I have not tried more than one topic.
Thanks

Re: how many topics spark streaming can handle

Posted by Michael Armbrust <mi...@databricks.com>.
I don't think that there is really a Spark specific limit here.  It would
be a function of the size of your spark / kafka clusters and the type of
processing you are trying to do.

On Mon, Jun 19, 2017 at 12:00 PM, Ashok Kumar <as...@yahoo.com.invalid>
wrote:

> Hi Gurus,
>
> Within one Spark streaming process how many topics can be handled? I have
> not tried more than one topic.
>
> Thanks
>