You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Charles Chen <cc...@google.com> on 2018/08/30 08:48:18 UTC
Re: Proposal for Beam Python User State and Timer APIs
Another update: the reference DirectRunner implementation of the Python
user state and timers API is out for review:
https://github.com/apache/beam/pull/6304
On Mon, Jul 9, 2018 at 2:18 PM Charles Chen <cc...@google.com> wrote:
> An update: https://github.com/apache/beam/pull/5691 has been merged. I
> hope to send out a reference implementation in the DirectRunner soon. On
> the roadmap after that is work on the relevant portability interfaces here
> so we can get this working on runners like Beam Python on Flink.
>
> On Wed, Jun 20, 2018 at 10:00 AM Charles Chen <cc...@google.com> wrote:
>
>> An update on the implementation: I recently sent out the user-facing
>> pipeline construction part of the API implementation out for review:
>> https://github.com/apache/beam/pull/5691.
>>
>> On Tue, Jun 5, 2018 at 5:26 PM Charles Chen <cc...@google.com> wrote:
>>
>>> Thanks everyone for contributing here. We've reached rough consensus on
>>> the approach we should take with this API, and I've summarized this in the
>>> new "Community consensus" sections I added to the doc (
>>> https://s.apache.org/beam-python-user-state-and-timers). I will begin
>>> initial implementation of this API soon.
>>>
>>> On Wed, May 23, 2018 at 8:08 PM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Nice proposal; it's exciting to see this about to be added to the SDK
>>>> as it enables a set of more complex use cases.
>>>>
>>>> I also think that some of the content can later be repurposed as user
>>>> documentation.
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>> On Wed, May 23, 2018 at 11:49 AM, Charles Chen <cc...@google.com> wrote:
>>>>
>>>>> Thanks everyone for the detailed comments and discussions. It looks
>>>>> like by now, we mostly agree with the requirements and overall direction
>>>>> needed for the API, though there is continuing discussion on specific
>>>>> details. I want to highlight two new sections of the doc, which address
>>>>> some discussions that have come up:
>>>>>
>>>>> - *Existing state and transactionality*: this section addresses
>>>>> how we will address an existing transactionality inconsistency in the
>>>>> existing Java API. (
>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ofyl9jspiz3b
>>>>> )
>>>>> - *State for merging windows*: this section addresses how we will
>>>>> deal with non-combinable state in conjunction with merging windows. (
>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ctxkcgabtzpy
>>>>> )
>>>>>
>>>>> Let me know any further comments and suggestions.
>>>>>
>>>>> On Tue, May 22, 2018 at 9:29 AM Kenneth Knowles <kl...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Nice. I know that Java users have found it helpful to have this
>>>>>> lower-level way of writing pipelines when the high-level primitives don't
>>>>>> quite have the tight control they are looking for. I hope it will be a big
>>>>>> draw for Python, too.
>>>>>>
>>>>>> (commenting on the doc)
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Mon, May 21, 2018 at 5:15 PM Charles Chen <cc...@google.com> wrote:
>>>>>>
>>>>>>> I want to share a proposal for adding user state and timer support
>>>>>>> to the Beam Python SDK and get the community's thoughts on how such an API
>>>>>>> should look: https://s.apache.org/beam-python-user-state-and-timers
>>>>>>>
>>>>>>> Let me know what you think and please add any comments and
>>>>>>> suggestions you may have.
>>>>>>>
>>>>>>> Best,
>>>>>>> Charles
>>>>>>>
>>>>>>
>>>>
***UNCHECKED*** Re: Proposal for Beam Python User State and Timer
APIs
Posted by Robert Bradshaw <ro...@google.com>.
And its implementation of the Fn API is on it's way too:
https://github.com/apache/beam/pull/6349
https://github.com/apache/beam/pull/6433
On Tue, Sep 18, 2018 at 6:56 PM Charles Chen <cc...@google.com> wrote:
> An update: the reference DirectRunner implementation of (and common
> execution code for) the Python user state and timers API has been merged:
> https://github.com/apache/beam/pull/6304
>
> On Thu, Aug 30, 2018 at 1:48 AM Charles Chen <cc...@google.com> wrote:
>
>> Another update: the reference DirectRunner implementation of the Python
>> user state and timers API is out for review:
>> https://github.com/apache/beam/pull/6304
>>
>> On Mon, Jul 9, 2018 at 2:18 PM Charles Chen <cc...@google.com> wrote:
>>
>>> An update: https://github.com/apache/beam/pull/5691 has been merged. I
>>> hope to send out a reference implementation in the DirectRunner soon. On
>>> the roadmap after that is work on the relevant portability interfaces here
>>> so we can get this working on runners like Beam Python on Flink.
>>>
>>> On Wed, Jun 20, 2018 at 10:00 AM Charles Chen <cc...@google.com> wrote:
>>>
>>>> An update on the implementation: I recently sent out the user-facing
>>>> pipeline construction part of the API implementation out for review:
>>>> https://github.com/apache/beam/pull/5691.
>>>>
>>>> On Tue, Jun 5, 2018 at 5:26 PM Charles Chen <cc...@google.com> wrote:
>>>>
>>>>> Thanks everyone for contributing here. We've reached rough consensus
>>>>> on the approach we should take with this API, and I've summarized this in
>>>>> the new "Community consensus" sections I added to the doc (
>>>>> https://s.apache.org/beam-python-user-state-and-timers). I will
>>>>> begin initial implementation of this API soon.
>>>>>
>>>>> On Wed, May 23, 2018 at 8:08 PM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Nice proposal; it's exciting to see this about to be added to the SDK
>>>>>> as it enables a set of more complex use cases.
>>>>>>
>>>>>> I also think that some of the content can later be repurposed as user
>>>>>> documentation.
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>> On Wed, May 23, 2018 at 11:49 AM, Charles Chen <cc...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks everyone for the detailed comments and discussions. It looks
>>>>>>> like by now, we mostly agree with the requirements and overall direction
>>>>>>> needed for the API, though there is continuing discussion on specific
>>>>>>> details. I want to highlight two new sections of the doc, which address
>>>>>>> some discussions that have come up:
>>>>>>>
>>>>>>> - *Existing state and transactionality*: this section addresses
>>>>>>> how we will address an existing transactionality inconsistency in the
>>>>>>> existing Java API. (
>>>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ofyl9jspiz3b
>>>>>>> )
>>>>>>> - *State for merging windows*: this section addresses how we
>>>>>>> will deal with non-combinable state in conjunction with merging windows. (
>>>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ctxkcgabtzpy
>>>>>>> )
>>>>>>>
>>>>>>> Let me know any further comments and suggestions.
>>>>>>>
>>>>>>> On Tue, May 22, 2018 at 9:29 AM Kenneth Knowles <kl...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Nice. I know that Java users have found it helpful to have this
>>>>>>>> lower-level way of writing pipelines when the high-level primitives don't
>>>>>>>> quite have the tight control they are looking for. I hope it will be a big
>>>>>>>> draw for Python, too.
>>>>>>>>
>>>>>>>> (commenting on the doc)
>>>>>>>>
>>>>>>>> Kenn
>>>>>>>>
>>>>>>>> On Mon, May 21, 2018 at 5:15 PM Charles Chen <cc...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I want to share a proposal for adding user state and timer support
>>>>>>>>> to the Beam Python SDK and get the community's thoughts on how such an API
>>>>>>>>> should look:
>>>>>>>>> https://s.apache.org/beam-python-user-state-and-timers
>>>>>>>>>
>>>>>>>>> Let me know what you think and please add any comments and
>>>>>>>>> suggestions you may have.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Charles
>>>>>>>>>
>>>>>>>>
>>>>>>
Re: Proposal for Beam Python User State and Timer APIs
Posted by Pablo Estrada <pa...@google.com>.
Very cool! Thanks Charles!
On Tue, Sep 18, 2018, 9:56 AM Charles Chen <cc...@google.com> wrote:
> An update: the reference DirectRunner implementation of (and common
> execution code for) the Python user state and timers API has been merged:
> https://github.com/apache/beam/pull/6304
>
> On Thu, Aug 30, 2018 at 1:48 AM Charles Chen <cc...@google.com> wrote:
>
>> Another update: the reference DirectRunner implementation of the Python
>> user state and timers API is out for review:
>> https://github.com/apache/beam/pull/6304
>>
>> On Mon, Jul 9, 2018 at 2:18 PM Charles Chen <cc...@google.com> wrote:
>>
>>> An update: https://github.com/apache/beam/pull/5691 has been merged. I
>>> hope to send out a reference implementation in the DirectRunner soon. On
>>> the roadmap after that is work on the relevant portability interfaces here
>>> so we can get this working on runners like Beam Python on Flink.
>>>
>>> On Wed, Jun 20, 2018 at 10:00 AM Charles Chen <cc...@google.com> wrote:
>>>
>>>> An update on the implementation: I recently sent out the user-facing
>>>> pipeline construction part of the API implementation out for review:
>>>> https://github.com/apache/beam/pull/5691.
>>>>
>>>> On Tue, Jun 5, 2018 at 5:26 PM Charles Chen <cc...@google.com> wrote:
>>>>
>>>>> Thanks everyone for contributing here. We've reached rough consensus
>>>>> on the approach we should take with this API, and I've summarized this in
>>>>> the new "Community consensus" sections I added to the doc (
>>>>> https://s.apache.org/beam-python-user-state-and-timers). I will
>>>>> begin initial implementation of this API soon.
>>>>>
>>>>> On Wed, May 23, 2018 at 8:08 PM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Nice proposal; it's exciting to see this about to be added to the SDK
>>>>>> as it enables a set of more complex use cases.
>>>>>>
>>>>>> I also think that some of the content can later be repurposed as user
>>>>>> documentation.
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>> On Wed, May 23, 2018 at 11:49 AM, Charles Chen <cc...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks everyone for the detailed comments and discussions. It looks
>>>>>>> like by now, we mostly agree with the requirements and overall direction
>>>>>>> needed for the API, though there is continuing discussion on specific
>>>>>>> details. I want to highlight two new sections of the doc, which address
>>>>>>> some discussions that have come up:
>>>>>>>
>>>>>>> - *Existing state and transactionality*: this section addresses
>>>>>>> how we will address an existing transactionality inconsistency in the
>>>>>>> existing Java API. (
>>>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ofyl9jspiz3b
>>>>>>> )
>>>>>>> - *State for merging windows*: this section addresses how we
>>>>>>> will deal with non-combinable state in conjunction with merging windows. (
>>>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ctxkcgabtzpy
>>>>>>> )
>>>>>>>
>>>>>>> Let me know any further comments and suggestions.
>>>>>>>
>>>>>>> On Tue, May 22, 2018 at 9:29 AM Kenneth Knowles <kl...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Nice. I know that Java users have found it helpful to have this
>>>>>>>> lower-level way of writing pipelines when the high-level primitives don't
>>>>>>>> quite have the tight control they are looking for. I hope it will be a big
>>>>>>>> draw for Python, too.
>>>>>>>>
>>>>>>>> (commenting on the doc)
>>>>>>>>
>>>>>>>> Kenn
>>>>>>>>
>>>>>>>> On Mon, May 21, 2018 at 5:15 PM Charles Chen <cc...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I want to share a proposal for adding user state and timer support
>>>>>>>>> to the Beam Python SDK and get the community's thoughts on how such an API
>>>>>>>>> should look:
>>>>>>>>> https://s.apache.org/beam-python-user-state-and-timers
>>>>>>>>>
>>>>>>>>> Let me know what you think and please add any comments and
>>>>>>>>> suggestions you may have.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Charles
>>>>>>>>>
>>>>>>>>
>>>>>>
Re: Proposal for Beam Python User State and Timer APIs
Posted by Charles Chen <cc...@google.com>.
An update: the reference DirectRunner implementation of (and common
execution code for) the Python user state and timers API has been merged:
https://github.com/apache/beam/pull/6304
On Thu, Aug 30, 2018 at 1:48 AM Charles Chen <cc...@google.com> wrote:
> Another update: the reference DirectRunner implementation of the Python
> user state and timers API is out for review:
> https://github.com/apache/beam/pull/6304
>
> On Mon, Jul 9, 2018 at 2:18 PM Charles Chen <cc...@google.com> wrote:
>
>> An update: https://github.com/apache/beam/pull/5691 has been merged. I
>> hope to send out a reference implementation in the DirectRunner soon. On
>> the roadmap after that is work on the relevant portability interfaces here
>> so we can get this working on runners like Beam Python on Flink.
>>
>> On Wed, Jun 20, 2018 at 10:00 AM Charles Chen <cc...@google.com> wrote:
>>
>>> An update on the implementation: I recently sent out the user-facing
>>> pipeline construction part of the API implementation out for review:
>>> https://github.com/apache/beam/pull/5691.
>>>
>>> On Tue, Jun 5, 2018 at 5:26 PM Charles Chen <cc...@google.com> wrote:
>>>
>>>> Thanks everyone for contributing here. We've reached rough consensus
>>>> on the approach we should take with this API, and I've summarized this in
>>>> the new "Community consensus" sections I added to the doc (
>>>> https://s.apache.org/beam-python-user-state-and-timers). I will begin
>>>> initial implementation of this API soon.
>>>>
>>>> On Wed, May 23, 2018 at 8:08 PM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Nice proposal; it's exciting to see this about to be added to the SDK
>>>>> as it enables a set of more complex use cases.
>>>>>
>>>>> I also think that some of the content can later be repurposed as user
>>>>> documentation.
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>> On Wed, May 23, 2018 at 11:49 AM, Charles Chen <cc...@google.com> wrote:
>>>>>
>>>>>> Thanks everyone for the detailed comments and discussions. It looks
>>>>>> like by now, we mostly agree with the requirements and overall direction
>>>>>> needed for the API, though there is continuing discussion on specific
>>>>>> details. I want to highlight two new sections of the doc, which address
>>>>>> some discussions that have come up:
>>>>>>
>>>>>> - *Existing state and transactionality*: this section addresses
>>>>>> how we will address an existing transactionality inconsistency in the
>>>>>> existing Java API. (
>>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ofyl9jspiz3b
>>>>>> )
>>>>>> - *State for merging windows*: this section addresses how we will
>>>>>> deal with non-combinable state in conjunction with merging windows. (
>>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ctxkcgabtzpy
>>>>>> )
>>>>>>
>>>>>> Let me know any further comments and suggestions.
>>>>>>
>>>>>> On Tue, May 22, 2018 at 9:29 AM Kenneth Knowles <kl...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Nice. I know that Java users have found it helpful to have this
>>>>>>> lower-level way of writing pipelines when the high-level primitives don't
>>>>>>> quite have the tight control they are looking for. I hope it will be a big
>>>>>>> draw for Python, too.
>>>>>>>
>>>>>>> (commenting on the doc)
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>> On Mon, May 21, 2018 at 5:15 PM Charles Chen <cc...@google.com> wrote:
>>>>>>>
>>>>>>>> I want to share a proposal for adding user state and timer support
>>>>>>>> to the Beam Python SDK and get the community's thoughts on how such an API
>>>>>>>> should look: https://s.apache.org/beam-python-user-state-and-timers
>>>>>>>>
>>>>>>>> Let me know what you think and please add any comments and
>>>>>>>> suggestions you may have.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Charles
>>>>>>>>
>>>>>>>
>>>>>