You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Ofir Manor <of...@equalum.io> on 2016/05/11 08:47:55 UTC

Structured Streaming with Kafka source/sink

Hi,
I'm trying out Structured Streaming from current 2.0 branch.
Does the branch currently support Kafka as either source or sink? I
couldn't find a specific JIRA or design doc for that in SPARK-8360 or in
the examples... Is it still targeted for 2.0?

Also, I naively assume it will look similar to hdfs or JDBC *stream("path")*,
where the path will be some sort of Kafka URI (maybe protocol + broker list
+ topic list). Is that the current thinking?

Thanks,

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io

Re: Structured Streaming with Kafka source/sink

Posted by Ted Yu <yu...@gmail.com>.

Please see this thread:

http://search-hadoop.com/m/q3RTt9XAz651PiG/Adhoc+queries+spark+streaming&subj=Re+Adhoc+queries+on+Spark+2+0+with+Structured+Streaming

> On May 11, 2016, at 1:47 AM, Ofir Manor <of...@equalum.io> wrote:
> 
> Hi,
> I'm trying out Structured Streaming from current 2.0 branch.
> Does the branch currently support Kafka as either source or sink? I couldn't find a specific JIRA or design doc for that in SPARK-8360 or in the examples... Is it still targeted for 2.0?
> 
> Also, I naively assume it will look similar to hdfs or JDBC stream("path"), where the path will be some sort of Kafka URI (maybe protocol + broker list + topic list). Is that the current thinking?
> 
> Thanks,
> Ofir Manor
> 
> Co-Founder & CTO | Equalum
> 
> Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io