You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Yu Zhang <yz...@gmail.com> on 2021/05/20 18:37:10 UTC

Spark Structured Streaming Runner Roadmap

Hi Beam Community, 

Would there be any roadmap for Spark Structured Runner to support streaming and Splittable DoFn API? Like the specific timeline or release version. 

Thanks, 
Yu

Re: Spark Structured Streaming Runner Roadmap

Posted by Etienne Chauchot <ec...@apache.org>.

Hi,

Sorry for the late answer: the streaming mode in spark structured 
streaming runner is stuck because of spark structured streaming 
framework implementation of watermark at the apache spark project side. See

https://echauchot.blogspot.com/2020/11/watermark-architecture-proposal-for.html

best

Etienne

On 20/05/2021 20:37, Yu Zhang wrote:
> Hi Beam Community,
>
> Would there be any roadmap for Spark Structured Runner to support streaming and Splittable DoFn API? Like the specific timeline or release version.
>
> Thanks,
> Yu

Re: Spark Structured Streaming Runner Roadmap

Posted by Etienne Chauchot <ec...@apache.org>.

Hi,

Sorry for the late answer: the streaming mode in spark structured 
streaming runner is stuck because of spark structured streaming 
framework implementation of watermark at the apache spark project side. See

https://echauchot.blogspot.com/2020/11/watermark-architecture-proposal-for.html

best

Etienne

On 20/05/2021 20:37, Yu Zhang wrote:
> Hi Beam Community,
>
> Would there be any roadmap for Spark Structured Runner to support streaming and Splittable DoFn API? Like the specific timeline or release version.
>
> Thanks,
> Yu