You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "A.W. Covert III" <co...@gmail.com> on 2016/07/20 20:22:38 UTC

PySpark 2.0 Structured Streaming Question

Hi All,

I've been digging into spark 2.0, I have some streaming jobs running well
on YARN, and I'm working on some Spark Structured Streaming jobs now.

I have a couple of jobs I'd like to move to Structured Streaming with the
`foreachWriter` but it's not available in PySpark yet. Is it just because
Structured Streaming is still in alpha or is there some technical subtlety
that I'm overlooking?

Thanks,

Art Covert

http://covertphotography.tumblr.com/
https://github.com/covertspartan/

Re: PySpark 2.0 Structured Streaming Question

Posted by Tathagata Das <ta...@gmail.com>.

foreachWriter is not currently available in the python. we dont have a
clear plan yet on when foreachWriter will be available in Python.

On Wed, Jul 20, 2016 at 1:22 PM, A.W. Covert III <co...@gmail.com> wrote:

> Hi All,
>
> I've been digging into spark 2.0, I have some streaming jobs running well
> on YARN, and I'm working on some Spark Structured Streaming jobs now.
>
> I have a couple of jobs I'd like to move to Structured Streaming with the
> `foreachWriter` but it's not available in PySpark yet. Is it just because
> Structured Streaming is still in alpha or is there some technical subtlety
> that I'm overlooking?
>
> Thanks,
>
> Art Covert
>
> http://covertphotography.tumblr.com/
> https://github.com/covertspartan/
>
>
>