You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Manohar753 <ma...@happiestminds.com> on 2015/06/19 10:56:52 UTC

N kafka topics vs N spark Streaming

Hi Everybody,

I have four kafks topics each for
separateoperation(Add,Delete,Update,Merge).
so spark also will have four consumed streams,so how we can run my spark job
here?

should i run four spark jobs separately?
is there any way to bundle all streams  into singlejar and run as single
Job?

Thanks in Advance.





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/N-kafka-topics-vs-N-spark-Streaming-tp23408.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: N kafka topics vs N spark Streaming

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

Like this?

val add_msgs = KafkaUtils.createDirectStream[String, String, StringDecoder,
StringDecoder](
      ssc, kafkaParams, Array("add").toSet)

val delete_msgs = KafkaUtils.createDirectStream[String, String,
StringDecoder, StringDecoder](
      ssc, kafkaParams, Array("delete").toSet)

val update_msgs = KafkaUtils.createDirectStream[String, String,
StringDecoder, StringDecoder](
      ssc, kafkaParams, Array("update").toSet)

val merge_msgs = KafkaUtils.createDirectStream[String, String,
StringDecoder, StringDecoder](
      ssc, kafkaParams, Array("merge").toSet)


It should be fine if your batch duration is same for all. Now, if you want
a single stream with all data in it, then you can do like:

val all_msgs = KafkaUtils.createDirectStream[String, String, StringDecoder,
StringDecoder](
      ssc, kafkaParams, Array("delete","add","update","merge").toSet)

Thanks
Best Regards

On Fri, Jun 19, 2015 at 2:26 PM, Manohar753 <manohar.reddy@happiestminds.com
> wrote:

> Hi Everybody,
>
> I have four kafks topics each for
> separateoperation(Add,Delete,Update,Merge).
> so spark also will have four consumed streams,so how we can run my spark
> job
> here?
>
> should i run four spark jobs separately?
> is there any way to bundle all streams  into singlejar and run as single
> Job?
>
> Thanks in Advance.
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/N-kafka-topics-vs-N-spark-Streaming-tp23408.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>