You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2019/08/02 22:56:52 UTC

Python API for mapGroupsWithState

Can someone succinctly describe the challenge in adding the
`mapGroupsWithState()` API to PySpark?

I was hoping for some suboptimal but nonetheless working solution to be
available in Python, as there are with Python UDFs for example, but that
doesn't seem to be case. The JIRA ticket for arbitrary stateful operations
in Structured Streaming <https://issues.apache.org/jira/browse/SPARK-19067>
doesn't give any indication that a Python version of the API is coming.

Is this something that will likely be added in the near future, or is it a
major undertaking? Can someone briefly describe the problem?

Nick

Re: Python API for mapGroupsWithState

Posted by Stavros Kontopoulos <st...@lightbend.com>.
+1 I was looking at this today, so any idea why this was not added before?

On Sat, Aug 3, 2019 at 1:57 AM Nicholas Chammas <ni...@gmail.com>
wrote:

> Can someone succinctly describe the challenge in adding the
> `mapGroupsWithState()` API to PySpark?
>
> I was hoping for some suboptimal but nonetheless working solution to be
> available in Python, as there are with Python UDFs for example, but that
> doesn't seem to be case. The JIRA ticket for arbitrary stateful
> operations in Structured Streaming
> <https://issues.apache.org/jira/browse/SPARK-19067> doesn't give any
> indication that a Python version of the API is coming.
>
> Is this something that will likely be added in the near future, or is it a
> major undertaking? Can someone briefly describe the problem?
>
> Nick
>
>

-- 
Stavros Kontopoulos
*Principal Engineer*
*Lightbend Platform <https://www.lightbend.com/lightbend-platform>*
*mob: **+30 6977967274 <+30+6977967274>*