You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Maximilian Michels <mx...@apache.org> on 2019/04/11 12:00:04 UTC
[docs] Python State & Timers
Hi everyone,
The Python SDK still lacks documentation on state and timers.
As a first step, what do you think about updating these two blog posts
with the corresponding Python code?
https://beam.apache.org/blog/2017/02/13/stateful-processing.html
https://beam.apache.org/blog/2017/08/28/timely-processing.html
Thanks,
Max
Re: [docs] Python State & Timers
Posted by Brian Hulette <bh...@google.com>.
Thanks for the great discussion on this! I hadn't even thought about the
potential race condition for ValueState. It does seem like we should be
encouraging people to use a CombiningState if at all possible.
It sounds like this thread is resolved (we know how to add python examples
to the stateful/timely blog posts) but a new one about either adding
ValueState to python or removing it from Java is warranted.
On Thu, Apr 25, 2019 at 8:31 AM Robert Bradshaw <ro...@google.com> wrote:
>
>
> On Thu, Apr 25, 2019, 5:26 PM Maximilian Michels <mx...@apache.org> wrote:
>
>> Completely agree that CombiningState is nicer in this example. Users may
>> still want to use ValueState when there is nothing to combine.
>
>
> I've always had trouble coming up with any good examples of this.
>
> Also,
>> users already know ValueState from the Java SDK.
>>
>
> Maybe we should deprecate that :)
>
>
> On 25.04.19 17:12, Robert Bradshaw wrote:
>> > On Thu, Apr 25, 2019 at 4:58 PM Maximilian Michels <mx...@apache.org>
>> wrote:
>> >>
>> >> I forgot to give an example, just to clarify for others:
>> >>
>> >>> What was the specific example that was less natural?
>> >>
>> >> Basically every time we use ListState to express ValueState, e.g.
>> >>
>> >> next_index, = list(state.read()) or [0]
>> >>
>> >> Taken from:
>> >>
>> https://github.com/apache/beam/pull/8363/files#diff-ba1a2aed98079ccce869cd660ca9d97dR301
>> >
>> > Yes, ListState is much less natural here. I think generally
>> > CombiningValue is often a better replacement. E.g. the Java example
>> > reads
>> >
>> >
>> > public void processElement(
>> > ProcessContext context, @StateId("index") ValueState<Integer>
>> index) {
>> > int current = firstNonNull(index.read(), 0);
>> > context.output(KV.of(current, context.element()));
>> > index.write(current+1);
>> > }
>> >
>> >
>> > which is replaced with bag state
>> >
>> >
>> > def process(self, element, state=DoFn.StateParam(INDEX_STATE)):
>> > next_index, = list(state.read()) or [0]
>> > yield (element, next_index)
>> > state.clear()
>> > state.add(next_index + 1)
>> >
>> >
>> > whereas CombiningState would be more natural (than ListState, and
>> > arguably than even ValueState), giving
>> >
>> >
>> > def process(self, element, index=DoFn.StateParam(INDEX_STATE)):
>> > yield element, index.read()
>> > index.add(1)
>> >
>> >
>> >
>> >
>> >>
>> >> -Max
>> >>
>> >> On 25.04.19 16:40, Robert Bradshaw wrote:
>> >>> https://github.com/apache/beam/pull/8402
>> >>>
>> >>> On Thu, Apr 25, 2019 at 4:26 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>>>
>> >>>> Oh, this is for the indexing example.
>> >>>>
>> >>>> I actually think using CombiningState is more cleaner than
>> ValueState.
>> >>>>
>> >>>>
>> https://github.com/apache/beam/blob/release-2.12.0/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py#L262
>> >>>>
>> >>>> (The fact that one must specify the accumulator coder is, however,
>> >>>> unfortunate. We should probably infer that if we can.)
>> >>>>
>> >>>> On Thu, Apr 25, 2019 at 4:19 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>>>>
>> >>>>> The desire was to avoid the implicit disallowed combination wart in
>> >>>>> Python (until we could make sense of it), and also ValueState could
>> be
>> >>>>> surprising with respect to older values overwriting newer ones. What
>> >>>>> was the specific example that was less natural?
>> >>>>>
>> >>>>> On Thu, Apr 25, 2019 at 3:01 PM Maximilian Michels <mx...@apache.org>
>> wrote:
>> >>>>>>
>> >>>>>> @Pablo: Thanks for following up with the PR! :)
>> >>>>>>
>> >>>>>> @Brian: I was wondering about this as well. It makes the Python
>> state
>> >>>>>> code a bit unnatural. I'd suggest to add a ValueState wrapper
>> around
>> >>>>>> ListState/CombiningState.
>> >>>>>>
>> >>>>>> @Robert: Like Reuven pointed out, we can disallow ValueState for
>> merging
>> >>>>>> windows with state.
>> >>>>>>
>> >>>>>> @Reza: Great. Let's make sure it has Python examples out of the
>> box.
>> >>>>>> Either Pablo or me could help there.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Max
>> >>>>>>
>> >>>>>> On 25.04.19 04:14, Reza Ardeshir Rokni wrote:
>> >>>>>>> Pablo, Kenneth and I have a new blog ready for publication which
>> covers
>> >>>>>>> how to create a "looping timer" it allows for default values to be
>> >>>>>>> created in a window when no incoming elements exists. We just
>> need to
>> >>>>>>> clear a few bits before publication, but would be great to have
>> that
>> >>>>>>> also include a python example, I wrote it in java...
>> >>>>>>>
>> >>>>>>> Cheers
>> >>>>>>>
>> >>>>>>> Reza
>> >>>>>>>
>> >>>>>>> On Thu, 25 Apr 2019 at 04:34, Reuven Lax <relax@google.com
>> >>>>>>> <ma...@google.com>> wrote:
>> >>>>>>>
>> >>>>>>> Well state is still not implemented for merging windows
>> even for
>> >>>>>>> Java (though I believe the idea was to disallow ValueState
>> there).
>> >>>>>>>
>> >>>>>>> On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <
>> robertwb@google.com
>> >>>>>>> <ma...@google.com>> wrote:
>> >>>>>>>
>> >>>>>>> It was unclear what the semantics were for ValueState
>> for merging
>> >>>>>>> windows. (It's also a bit weird as it's inherently a
>> race condition
>> >>>>>>> wrt element ordering, unlike Bag and CombineState,
>> though you can
>> >>>>>>> always implement it as a CombineState that always
>> returns the latest
>> >>>>>>> value which is a bit more explicit about the dangers
>> here.)
>> >>>>>>>
>> >>>>>>> On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette
>> >>>>>>> <bhulette@google.com <ma...@google.com>>
>> wrote:
>> >>>>>>> >
>> >>>>>>> > That's a great idea! I thought about this too after
>> those
>> >>>>>>> posts came up on the list recently. I started to look
>> into it,
>> >>>>>>> but I noticed that there's actually no implementation of
>> >>>>>>> ValueState in userstate. Is there a reason for that? I
>> started
>> >>>>>>> to work on a patch to add it but I was just curious if
>> there was
>> >>>>>>> some reason it was omitted that I should be aware of.
>> >>>>>>> >
>> >>>>>>> > We could certainly replicate the example without
>> ValueState
>> >>>>>>> by using BagState and clearing it before each write,
>> but it
>> >>>>>>> would be nice if we could draw a direct parallel.
>> >>>>>>> >
>> >>>>>>> > Brian
>> >>>>>>> >
>> >>>>>>> > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels
>> >>>>>>> <mxm@apache.org <ma...@apache.org>> wrote:
>> >>>>>>> >>
>> >>>>>>> >> > It would probably be pretty easy to add the
>> corresponding
>> >>>>>>> code snippets to the docs as well.
>> >>>>>>> >>
>> >>>>>>> >> It's probably a bit more work because there is no
>> section
>> >>>>>>> dedicated to
>> >>>>>>> >> state/timer yet in the documentation. Tracked here:
>> >>>>>>> >> https://jira.apache.org/jira/browse/BEAM-2472
>> >>>>>>> >>
>> >>>>>>> >> > I've been going over this topic a bit. I'll add
>> the
>> >>>>>>> snippets next week, if that's fine by y'all.
>> >>>>>>> >>
>> >>>>>>> >> That would be great. The blog posts are a great way
>> to get
>> >>>>>>> started with
>> >>>>>>> >> state/timers.
>> >>>>>>> >>
>> >>>>>>> >> Thanks,
>> >>>>>>> >> Max
>> >>>>>>> >>
>> >>>>>>> >> On 11.04.19 20:21, Pablo Estrada wrote:
>> >>>>>>> >> > I've been going over this topic a bit. I'll add
>> the
>> >>>>>>> snippets next week,
>> >>>>>>> >> > if that's fine by y'all.
>> >>>>>>> >> > Best
>> >>>>>>> >> > -P.
>> >>>>>>> >> >
>> >>>>>>> >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw
>> >>>>>>> <robertwb@google.com <ma...@google.com>
>> >>>>>>> >> > <mailto:robertwb@google.com <mailto:
>> robertwb@google.com>>>
>> >>>>>>> wrote:
>> >>>>>>> >> >
>> >>>>>>> >> > That's a great idea! It would probably be
>> pretty easy
>> >>>>>>> to add the
>> >>>>>>> >> > corresponding code snippets to the docs as
>> well.
>> >>>>>>> >> >
>> >>>>>>> >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian
>> Michels
>> >>>>>>> <mxm@apache.org <ma...@apache.org>
>> >>>>>>> >> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> wrote:
>> >>>>>>> >> > >
>> >>>>>>> >> > > Hi everyone,
>> >>>>>>> >> > >
>> >>>>>>> >> > > The Python SDK still lacks documentation
>> on state
>> >>>>>>> and timers.
>> >>>>>>> >> > >
>> >>>>>>> >> > > As a first step, what do you think about
>> updating
>> >>>>>>> these two blog
>> >>>>>>> >> > posts
>> >>>>>>> >> > > with the corresponding Python code?
>> >>>>>>> >> > >
>> >>>>>>> >> > >
>> >>>>>>>
>> https://beam.apache.org/blog/2017/02/13/stateful-processing.html
>> >>>>>>> >> > >
>> >>>>>>>
>> https://beam.apache.org/blog/2017/08/28/timely-processing.html
>> >>>>>>> >> > >
>> >>>>>>> >> > > Thanks,
>> >>>>>>> >> > > Max
>> >>>>>>> >> >
>> >>>>>>>
>>
>
Re: [docs] Python State & Timers
Posted by Robert Bradshaw <ro...@google.com>.
On Thu, Apr 25, 2019, 5:26 PM Maximilian Michels <mx...@apache.org> wrote:
> Completely agree that CombiningState is nicer in this example. Users may
> still want to use ValueState when there is nothing to combine.
I've always had trouble coming up with any good examples of this.
Also,
> users already know ValueState from the Java SDK.
>
Maybe we should deprecate that :)
On 25.04.19 17:12, Robert Bradshaw wrote:
> > On Thu, Apr 25, 2019 at 4:58 PM Maximilian Michels <mx...@apache.org>
> wrote:
> >>
> >> I forgot to give an example, just to clarify for others:
> >>
> >>> What was the specific example that was less natural?
> >>
> >> Basically every time we use ListState to express ValueState, e.g.
> >>
> >> next_index, = list(state.read()) or [0]
> >>
> >> Taken from:
> >>
> https://github.com/apache/beam/pull/8363/files#diff-ba1a2aed98079ccce869cd660ca9d97dR301
> >
> > Yes, ListState is much less natural here. I think generally
> > CombiningValue is often a better replacement. E.g. the Java example
> > reads
> >
> >
> > public void processElement(
> > ProcessContext context, @StateId("index") ValueState<Integer>
> index) {
> > int current = firstNonNull(index.read(), 0);
> > context.output(KV.of(current, context.element()));
> > index.write(current+1);
> > }
> >
> >
> > which is replaced with bag state
> >
> >
> > def process(self, element, state=DoFn.StateParam(INDEX_STATE)):
> > next_index, = list(state.read()) or [0]
> > yield (element, next_index)
> > state.clear()
> > state.add(next_index + 1)
> >
> >
> > whereas CombiningState would be more natural (than ListState, and
> > arguably than even ValueState), giving
> >
> >
> > def process(self, element, index=DoFn.StateParam(INDEX_STATE)):
> > yield element, index.read()
> > index.add(1)
> >
> >
> >
> >
> >>
> >> -Max
> >>
> >> On 25.04.19 16:40, Robert Bradshaw wrote:
> >>> https://github.com/apache/beam/pull/8402
> >>>
> >>> On Thu, Apr 25, 2019 at 4:26 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>>>
> >>>> Oh, this is for the indexing example.
> >>>>
> >>>> I actually think using CombiningState is more cleaner than ValueState.
> >>>>
> >>>>
> https://github.com/apache/beam/blob/release-2.12.0/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py#L262
> >>>>
> >>>> (The fact that one must specify the accumulator coder is, however,
> >>>> unfortunate. We should probably infer that if we can.)
> >>>>
> >>>> On Thu, Apr 25, 2019 at 4:19 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>>>>
> >>>>> The desire was to avoid the implicit disallowed combination wart in
> >>>>> Python (until we could make sense of it), and also ValueState could
> be
> >>>>> surprising with respect to older values overwriting newer ones. What
> >>>>> was the specific example that was less natural?
> >>>>>
> >>>>> On Thu, Apr 25, 2019 at 3:01 PM Maximilian Michels <mx...@apache.org>
> wrote:
> >>>>>>
> >>>>>> @Pablo: Thanks for following up with the PR! :)
> >>>>>>
> >>>>>> @Brian: I was wondering about this as well. It makes the Python
> state
> >>>>>> code a bit unnatural. I'd suggest to add a ValueState wrapper around
> >>>>>> ListState/CombiningState.
> >>>>>>
> >>>>>> @Robert: Like Reuven pointed out, we can disallow ValueState for
> merging
> >>>>>> windows with state.
> >>>>>>
> >>>>>> @Reza: Great. Let's make sure it has Python examples out of the box.
> >>>>>> Either Pablo or me could help there.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Max
> >>>>>>
> >>>>>> On 25.04.19 04:14, Reza Ardeshir Rokni wrote:
> >>>>>>> Pablo, Kenneth and I have a new blog ready for publication which
> covers
> >>>>>>> how to create a "looping timer" it allows for default values to be
> >>>>>>> created in a window when no incoming elements exists. We just need
> to
> >>>>>>> clear a few bits before publication, but would be great to have
> that
> >>>>>>> also include a python example, I wrote it in java...
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>>
> >>>>>>> Reza
> >>>>>>>
> >>>>>>> On Thu, 25 Apr 2019 at 04:34, Reuven Lax <relax@google.com
> >>>>>>> <ma...@google.com>> wrote:
> >>>>>>>
> >>>>>>> Well state is still not implemented for merging windows even
> for
> >>>>>>> Java (though I believe the idea was to disallow ValueState
> there).
> >>>>>>>
> >>>>>>> On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <
> robertwb@google.com
> >>>>>>> <ma...@google.com>> wrote:
> >>>>>>>
> >>>>>>> It was unclear what the semantics were for ValueState
> for merging
> >>>>>>> windows. (It's also a bit weird as it's inherently a
> race condition
> >>>>>>> wrt element ordering, unlike Bag and CombineState,
> though you can
> >>>>>>> always implement it as a CombineState that always
> returns the latest
> >>>>>>> value which is a bit more explicit about the dangers
> here.)
> >>>>>>>
> >>>>>>> On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette
> >>>>>>> <bhulette@google.com <ma...@google.com>>
> wrote:
> >>>>>>> >
> >>>>>>> > That's a great idea! I thought about this too after
> those
> >>>>>>> posts came up on the list recently. I started to look
> into it,
> >>>>>>> but I noticed that there's actually no implementation of
> >>>>>>> ValueState in userstate. Is there a reason for that? I
> started
> >>>>>>> to work on a patch to add it but I was just curious if
> there was
> >>>>>>> some reason it was omitted that I should be aware of.
> >>>>>>> >
> >>>>>>> > We could certainly replicate the example without
> ValueState
> >>>>>>> by using BagState and clearing it before each write, but
> it
> >>>>>>> would be nice if we could draw a direct parallel.
> >>>>>>> >
> >>>>>>> > Brian
> >>>>>>> >
> >>>>>>> > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels
> >>>>>>> <mxm@apache.org <ma...@apache.org>> wrote:
> >>>>>>> >>
> >>>>>>> >> > It would probably be pretty easy to add the
> corresponding
> >>>>>>> code snippets to the docs as well.
> >>>>>>> >>
> >>>>>>> >> It's probably a bit more work because there is no
> section
> >>>>>>> dedicated to
> >>>>>>> >> state/timer yet in the documentation. Tracked here:
> >>>>>>> >> https://jira.apache.org/jira/browse/BEAM-2472
> >>>>>>> >>
> >>>>>>> >> > I've been going over this topic a bit. I'll add the
> >>>>>>> snippets next week, if that's fine by y'all.
> >>>>>>> >>
> >>>>>>> >> That would be great. The blog posts are a great way
> to get
> >>>>>>> started with
> >>>>>>> >> state/timers.
> >>>>>>> >>
> >>>>>>> >> Thanks,
> >>>>>>> >> Max
> >>>>>>> >>
> >>>>>>> >> On 11.04.19 20:21, Pablo Estrada wrote:
> >>>>>>> >> > I've been going over this topic a bit. I'll add the
> >>>>>>> snippets next week,
> >>>>>>> >> > if that's fine by y'all.
> >>>>>>> >> > Best
> >>>>>>> >> > -P.
> >>>>>>> >> >
> >>>>>>> >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw
> >>>>>>> <robertwb@google.com <ma...@google.com>
> >>>>>>> >> > <mailto:robertwb@google.com <mailto:
> robertwb@google.com>>>
> >>>>>>> wrote:
> >>>>>>> >> >
> >>>>>>> >> > That's a great idea! It would probably be
> pretty easy
> >>>>>>> to add the
> >>>>>>> >> > corresponding code snippets to the docs as
> well.
> >>>>>>> >> >
> >>>>>>> >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian
> Michels
> >>>>>>> <mxm@apache.org <ma...@apache.org>
> >>>>>>> >> > <mailto:mxm@apache.org <ma...@apache.org>>>
> wrote:
> >>>>>>> >> > >
> >>>>>>> >> > > Hi everyone,
> >>>>>>> >> > >
> >>>>>>> >> > > The Python SDK still lacks documentation on
> state
> >>>>>>> and timers.
> >>>>>>> >> > >
> >>>>>>> >> > > As a first step, what do you think about
> updating
> >>>>>>> these two blog
> >>>>>>> >> > posts
> >>>>>>> >> > > with the corresponding Python code?
> >>>>>>> >> > >
> >>>>>>> >> > >
> >>>>>>>
> https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> >>>>>>> >> > >
> >>>>>>>
> https://beam.apache.org/blog/2017/08/28/timely-processing.html
> >>>>>>> >> > >
> >>>>>>> >> > > Thanks,
> >>>>>>> >> > > Max
> >>>>>>> >> >
> >>>>>>>
>
Re: [docs] Python State & Timers
Posted by Maximilian Michels <mx...@apache.org>.
Completely agree that CombiningState is nicer in this example. Users may
still want to use ValueState when there is nothing to combine. Also,
users already know ValueState from the Java SDK.
On 25.04.19 17:12, Robert Bradshaw wrote:
> On Thu, Apr 25, 2019 at 4:58 PM Maximilian Michels <mx...@apache.org> wrote:
>>
>> I forgot to give an example, just to clarify for others:
>>
>>> What was the specific example that was less natural?
>>
>> Basically every time we use ListState to express ValueState, e.g.
>>
>> next_index, = list(state.read()) or [0]
>>
>> Taken from:
>> https://github.com/apache/beam/pull/8363/files#diff-ba1a2aed98079ccce869cd660ca9d97dR301
>
> Yes, ListState is much less natural here. I think generally
> CombiningValue is often a better replacement. E.g. the Java example
> reads
>
>
> public void processElement(
> ProcessContext context, @StateId("index") ValueState<Integer> index) {
> int current = firstNonNull(index.read(), 0);
> context.output(KV.of(current, context.element()));
> index.write(current+1);
> }
>
>
> which is replaced with bag state
>
>
> def process(self, element, state=DoFn.StateParam(INDEX_STATE)):
> next_index, = list(state.read()) or [0]
> yield (element, next_index)
> state.clear()
> state.add(next_index + 1)
>
>
> whereas CombiningState would be more natural (than ListState, and
> arguably than even ValueState), giving
>
>
> def process(self, element, index=DoFn.StateParam(INDEX_STATE)):
> yield element, index.read()
> index.add(1)
>
>
>
>
>>
>> -Max
>>
>> On 25.04.19 16:40, Robert Bradshaw wrote:
>>> https://github.com/apache/beam/pull/8402
>>>
>>> On Thu, Apr 25, 2019 at 4:26 PM Robert Bradshaw <ro...@google.com> wrote:
>>>>
>>>> Oh, this is for the indexing example.
>>>>
>>>> I actually think using CombiningState is more cleaner than ValueState.
>>>>
>>>> https://github.com/apache/beam/blob/release-2.12.0/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py#L262
>>>>
>>>> (The fact that one must specify the accumulator coder is, however,
>>>> unfortunate. We should probably infer that if we can.)
>>>>
>>>> On Thu, Apr 25, 2019 at 4:19 PM Robert Bradshaw <ro...@google.com> wrote:
>>>>>
>>>>> The desire was to avoid the implicit disallowed combination wart in
>>>>> Python (until we could make sense of it), and also ValueState could be
>>>>> surprising with respect to older values overwriting newer ones. What
>>>>> was the specific example that was less natural?
>>>>>
>>>>> On Thu, Apr 25, 2019 at 3:01 PM Maximilian Michels <mx...@apache.org> wrote:
>>>>>>
>>>>>> @Pablo: Thanks for following up with the PR! :)
>>>>>>
>>>>>> @Brian: I was wondering about this as well. It makes the Python state
>>>>>> code a bit unnatural. I'd suggest to add a ValueState wrapper around
>>>>>> ListState/CombiningState.
>>>>>>
>>>>>> @Robert: Like Reuven pointed out, we can disallow ValueState for merging
>>>>>> windows with state.
>>>>>>
>>>>>> @Reza: Great. Let's make sure it has Python examples out of the box.
>>>>>> Either Pablo or me could help there.
>>>>>>
>>>>>> Thanks,
>>>>>> Max
>>>>>>
>>>>>> On 25.04.19 04:14, Reza Ardeshir Rokni wrote:
>>>>>>> Pablo, Kenneth and I have a new blog ready for publication which covers
>>>>>>> how to create a "looping timer" it allows for default values to be
>>>>>>> created in a window when no incoming elements exists. We just need to
>>>>>>> clear a few bits before publication, but would be great to have that
>>>>>>> also include a python example, I wrote it in java...
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> Reza
>>>>>>>
>>>>>>> On Thu, 25 Apr 2019 at 04:34, Reuven Lax <relax@google.com
>>>>>>> <ma...@google.com>> wrote:
>>>>>>>
>>>>>>> Well state is still not implemented for merging windows even for
>>>>>>> Java (though I believe the idea was to disallow ValueState there).
>>>>>>>
>>>>>>> On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <robertwb@google.com
>>>>>>> <ma...@google.com>> wrote:
>>>>>>>
>>>>>>> It was unclear what the semantics were for ValueState for merging
>>>>>>> windows. (It's also a bit weird as it's inherently a race condition
>>>>>>> wrt element ordering, unlike Bag and CombineState, though you can
>>>>>>> always implement it as a CombineState that always returns the latest
>>>>>>> value which is a bit more explicit about the dangers here.)
>>>>>>>
>>>>>>> On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette
>>>>>>> <bhulette@google.com <ma...@google.com>> wrote:
>>>>>>> >
>>>>>>> > That's a great idea! I thought about this too after those
>>>>>>> posts came up on the list recently. I started to look into it,
>>>>>>> but I noticed that there's actually no implementation of
>>>>>>> ValueState in userstate. Is there a reason for that? I started
>>>>>>> to work on a patch to add it but I was just curious if there was
>>>>>>> some reason it was omitted that I should be aware of.
>>>>>>> >
>>>>>>> > We could certainly replicate the example without ValueState
>>>>>>> by using BagState and clearing it before each write, but it
>>>>>>> would be nice if we could draw a direct parallel.
>>>>>>> >
>>>>>>> > Brian
>>>>>>> >
>>>>>>> > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels
>>>>>>> <mxm@apache.org <ma...@apache.org>> wrote:
>>>>>>> >>
>>>>>>> >> > It would probably be pretty easy to add the corresponding
>>>>>>> code snippets to the docs as well.
>>>>>>> >>
>>>>>>> >> It's probably a bit more work because there is no section
>>>>>>> dedicated to
>>>>>>> >> state/timer yet in the documentation. Tracked here:
>>>>>>> >> https://jira.apache.org/jira/browse/BEAM-2472
>>>>>>> >>
>>>>>>> >> > I've been going over this topic a bit. I'll add the
>>>>>>> snippets next week, if that's fine by y'all.
>>>>>>> >>
>>>>>>> >> That would be great. The blog posts are a great way to get
>>>>>>> started with
>>>>>>> >> state/timers.
>>>>>>> >>
>>>>>>> >> Thanks,
>>>>>>> >> Max
>>>>>>> >>
>>>>>>> >> On 11.04.19 20:21, Pablo Estrada wrote:
>>>>>>> >> > I've been going over this topic a bit. I'll add the
>>>>>>> snippets next week,
>>>>>>> >> > if that's fine by y'all.
>>>>>>> >> > Best
>>>>>>> >> > -P.
>>>>>>> >> >
>>>>>>> >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw
>>>>>>> <robertwb@google.com <ma...@google.com>
>>>>>>> >> > <mailto:robertwb@google.com <ma...@google.com>>>
>>>>>>> wrote:
>>>>>>> >> >
>>>>>>> >> > That's a great idea! It would probably be pretty easy
>>>>>>> to add the
>>>>>>> >> > corresponding code snippets to the docs as well.
>>>>>>> >> >
>>>>>>> >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels
>>>>>>> <mxm@apache.org <ma...@apache.org>
>>>>>>> >> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
>>>>>>> >> > >
>>>>>>> >> > > Hi everyone,
>>>>>>> >> > >
>>>>>>> >> > > The Python SDK still lacks documentation on state
>>>>>>> and timers.
>>>>>>> >> > >
>>>>>>> >> > > As a first step, what do you think about updating
>>>>>>> these two blog
>>>>>>> >> > posts
>>>>>>> >> > > with the corresponding Python code?
>>>>>>> >> > >
>>>>>>> >> > >
>>>>>>> https://beam.apache.org/blog/2017/02/13/stateful-processing.html
>>>>>>> >> > >
>>>>>>> https://beam.apache.org/blog/2017/08/28/timely-processing.html
>>>>>>> >> > >
>>>>>>> >> > > Thanks,
>>>>>>> >> > > Max
>>>>>>> >> >
>>>>>>>
Re: [docs] Python State & Timers
Posted by Robert Bradshaw <ro...@google.com>.
On Thu, Apr 25, 2019 at 4:58 PM Maximilian Michels <mx...@apache.org> wrote:
>
> I forgot to give an example, just to clarify for others:
>
> > What was the specific example that was less natural?
>
> Basically every time we use ListState to express ValueState, e.g.
>
> next_index, = list(state.read()) or [0]
>
> Taken from:
> https://github.com/apache/beam/pull/8363/files#diff-ba1a2aed98079ccce869cd660ca9d97dR301
Yes, ListState is much less natural here. I think generally
CombiningValue is often a better replacement. E.g. the Java example
reads
public void processElement(
ProcessContext context, @StateId("index") ValueState<Integer> index) {
int current = firstNonNull(index.read(), 0);
context.output(KV.of(current, context.element()));
index.write(current+1);
}
which is replaced with bag state
def process(self, element, state=DoFn.StateParam(INDEX_STATE)):
next_index, = list(state.read()) or [0]
yield (element, next_index)
state.clear()
state.add(next_index + 1)
whereas CombiningState would be more natural (than ListState, and
arguably than even ValueState), giving
def process(self, element, index=DoFn.StateParam(INDEX_STATE)):
yield element, index.read()
index.add(1)
>
> -Max
>
> On 25.04.19 16:40, Robert Bradshaw wrote:
> > https://github.com/apache/beam/pull/8402
> >
> > On Thu, Apr 25, 2019 at 4:26 PM Robert Bradshaw <ro...@google.com> wrote:
> >>
> >> Oh, this is for the indexing example.
> >>
> >> I actually think using CombiningState is more cleaner than ValueState.
> >>
> >> https://github.com/apache/beam/blob/release-2.12.0/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py#L262
> >>
> >> (The fact that one must specify the accumulator coder is, however,
> >> unfortunate. We should probably infer that if we can.)
> >>
> >> On Thu, Apr 25, 2019 at 4:19 PM Robert Bradshaw <ro...@google.com> wrote:
> >>>
> >>> The desire was to avoid the implicit disallowed combination wart in
> >>> Python (until we could make sense of it), and also ValueState could be
> >>> surprising with respect to older values overwriting newer ones. What
> >>> was the specific example that was less natural?
> >>>
> >>> On Thu, Apr 25, 2019 at 3:01 PM Maximilian Michels <mx...@apache.org> wrote:
> >>>>
> >>>> @Pablo: Thanks for following up with the PR! :)
> >>>>
> >>>> @Brian: I was wondering about this as well. It makes the Python state
> >>>> code a bit unnatural. I'd suggest to add a ValueState wrapper around
> >>>> ListState/CombiningState.
> >>>>
> >>>> @Robert: Like Reuven pointed out, we can disallow ValueState for merging
> >>>> windows with state.
> >>>>
> >>>> @Reza: Great. Let's make sure it has Python examples out of the box.
> >>>> Either Pablo or me could help there.
> >>>>
> >>>> Thanks,
> >>>> Max
> >>>>
> >>>> On 25.04.19 04:14, Reza Ardeshir Rokni wrote:
> >>>>> Pablo, Kenneth and I have a new blog ready for publication which covers
> >>>>> how to create a "looping timer" it allows for default values to be
> >>>>> created in a window when no incoming elements exists. We just need to
> >>>>> clear a few bits before publication, but would be great to have that
> >>>>> also include a python example, I wrote it in java...
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> Reza
> >>>>>
> >>>>> On Thu, 25 Apr 2019 at 04:34, Reuven Lax <relax@google.com
> >>>>> <ma...@google.com>> wrote:
> >>>>>
> >>>>> Well state is still not implemented for merging windows even for
> >>>>> Java (though I believe the idea was to disallow ValueState there).
> >>>>>
> >>>>> On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <robertwb@google.com
> >>>>> <ma...@google.com>> wrote:
> >>>>>
> >>>>> It was unclear what the semantics were for ValueState for merging
> >>>>> windows. (It's also a bit weird as it's inherently a race condition
> >>>>> wrt element ordering, unlike Bag and CombineState, though you can
> >>>>> always implement it as a CombineState that always returns the latest
> >>>>> value which is a bit more explicit about the dangers here.)
> >>>>>
> >>>>> On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette
> >>>>> <bhulette@google.com <ma...@google.com>> wrote:
> >>>>> >
> >>>>> > That's a great idea! I thought about this too after those
> >>>>> posts came up on the list recently. I started to look into it,
> >>>>> but I noticed that there's actually no implementation of
> >>>>> ValueState in userstate. Is there a reason for that? I started
> >>>>> to work on a patch to add it but I was just curious if there was
> >>>>> some reason it was omitted that I should be aware of.
> >>>>> >
> >>>>> > We could certainly replicate the example without ValueState
> >>>>> by using BagState and clearing it before each write, but it
> >>>>> would be nice if we could draw a direct parallel.
> >>>>> >
> >>>>> > Brian
> >>>>> >
> >>>>> > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels
> >>>>> <mxm@apache.org <ma...@apache.org>> wrote:
> >>>>> >>
> >>>>> >> > It would probably be pretty easy to add the corresponding
> >>>>> code snippets to the docs as well.
> >>>>> >>
> >>>>> >> It's probably a bit more work because there is no section
> >>>>> dedicated to
> >>>>> >> state/timer yet in the documentation. Tracked here:
> >>>>> >> https://jira.apache.org/jira/browse/BEAM-2472
> >>>>> >>
> >>>>> >> > I've been going over this topic a bit. I'll add the
> >>>>> snippets next week, if that's fine by y'all.
> >>>>> >>
> >>>>> >> That would be great. The blog posts are a great way to get
> >>>>> started with
> >>>>> >> state/timers.
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >> Max
> >>>>> >>
> >>>>> >> On 11.04.19 20:21, Pablo Estrada wrote:
> >>>>> >> > I've been going over this topic a bit. I'll add the
> >>>>> snippets next week,
> >>>>> >> > if that's fine by y'all.
> >>>>> >> > Best
> >>>>> >> > -P.
> >>>>> >> >
> >>>>> >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw
> >>>>> <robertwb@google.com <ma...@google.com>
> >>>>> >> > <mailto:robertwb@google.com <ma...@google.com>>>
> >>>>> wrote:
> >>>>> >> >
> >>>>> >> > That's a great idea! It would probably be pretty easy
> >>>>> to add the
> >>>>> >> > corresponding code snippets to the docs as well.
> >>>>> >> >
> >>>>> >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels
> >>>>> <mxm@apache.org <ma...@apache.org>
> >>>>> >> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >>>>> >> > >
> >>>>> >> > > Hi everyone,
> >>>>> >> > >
> >>>>> >> > > The Python SDK still lacks documentation on state
> >>>>> and timers.
> >>>>> >> > >
> >>>>> >> > > As a first step, what do you think about updating
> >>>>> these two blog
> >>>>> >> > posts
> >>>>> >> > > with the corresponding Python code?
> >>>>> >> > >
> >>>>> >> > >
> >>>>> https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> >>>>> >> > >
> >>>>> https://beam.apache.org/blog/2017/08/28/timely-processing.html
> >>>>> >> > >
> >>>>> >> > > Thanks,
> >>>>> >> > > Max
> >>>>> >> >
> >>>>>
Re: [docs] Python State & Timers
Posted by Maximilian Michels <mx...@apache.org>.
I forgot to give an example, just to clarify for others:
> What was the specific example that was less natural?
Basically every time we use ListState to express ValueState, e.g.
next_index, = list(state.read()) or [0]
Taken from:
https://github.com/apache/beam/pull/8363/files#diff-ba1a2aed98079ccce869cd660ca9d97dR301
-Max
On 25.04.19 16:40, Robert Bradshaw wrote:
> https://github.com/apache/beam/pull/8402
>
> On Thu, Apr 25, 2019 at 4:26 PM Robert Bradshaw <ro...@google.com> wrote:
>>
>> Oh, this is for the indexing example.
>>
>> I actually think using CombiningState is more cleaner than ValueState.
>>
>> https://github.com/apache/beam/blob/release-2.12.0/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py#L262
>>
>> (The fact that one must specify the accumulator coder is, however,
>> unfortunate. We should probably infer that if we can.)
>>
>> On Thu, Apr 25, 2019 at 4:19 PM Robert Bradshaw <ro...@google.com> wrote:
>>>
>>> The desire was to avoid the implicit disallowed combination wart in
>>> Python (until we could make sense of it), and also ValueState could be
>>> surprising with respect to older values overwriting newer ones. What
>>> was the specific example that was less natural?
>>>
>>> On Thu, Apr 25, 2019 at 3:01 PM Maximilian Michels <mx...@apache.org> wrote:
>>>>
>>>> @Pablo: Thanks for following up with the PR! :)
>>>>
>>>> @Brian: I was wondering about this as well. It makes the Python state
>>>> code a bit unnatural. I'd suggest to add a ValueState wrapper around
>>>> ListState/CombiningState.
>>>>
>>>> @Robert: Like Reuven pointed out, we can disallow ValueState for merging
>>>> windows with state.
>>>>
>>>> @Reza: Great. Let's make sure it has Python examples out of the box.
>>>> Either Pablo or me could help there.
>>>>
>>>> Thanks,
>>>> Max
>>>>
>>>> On 25.04.19 04:14, Reza Ardeshir Rokni wrote:
>>>>> Pablo, Kenneth and I have a new blog ready for publication which covers
>>>>> how to create a "looping timer" it allows for default values to be
>>>>> created in a window when no incoming elements exists. We just need to
>>>>> clear a few bits before publication, but would be great to have that
>>>>> also include a python example, I wrote it in java...
>>>>>
>>>>> Cheers
>>>>>
>>>>> Reza
>>>>>
>>>>> On Thu, 25 Apr 2019 at 04:34, Reuven Lax <relax@google.com
>>>>> <ma...@google.com>> wrote:
>>>>>
>>>>> Well state is still not implemented for merging windows even for
>>>>> Java (though I believe the idea was to disallow ValueState there).
>>>>>
>>>>> On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <robertwb@google.com
>>>>> <ma...@google.com>> wrote:
>>>>>
>>>>> It was unclear what the semantics were for ValueState for merging
>>>>> windows. (It's also a bit weird as it's inherently a race condition
>>>>> wrt element ordering, unlike Bag and CombineState, though you can
>>>>> always implement it as a CombineState that always returns the latest
>>>>> value which is a bit more explicit about the dangers here.)
>>>>>
>>>>> On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette
>>>>> <bhulette@google.com <ma...@google.com>> wrote:
>>>>> >
>>>>> > That's a great idea! I thought about this too after those
>>>>> posts came up on the list recently. I started to look into it,
>>>>> but I noticed that there's actually no implementation of
>>>>> ValueState in userstate. Is there a reason for that? I started
>>>>> to work on a patch to add it but I was just curious if there was
>>>>> some reason it was omitted that I should be aware of.
>>>>> >
>>>>> > We could certainly replicate the example without ValueState
>>>>> by using BagState and clearing it before each write, but it
>>>>> would be nice if we could draw a direct parallel.
>>>>> >
>>>>> > Brian
>>>>> >
>>>>> > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels
>>>>> <mxm@apache.org <ma...@apache.org>> wrote:
>>>>> >>
>>>>> >> > It would probably be pretty easy to add the corresponding
>>>>> code snippets to the docs as well.
>>>>> >>
>>>>> >> It's probably a bit more work because there is no section
>>>>> dedicated to
>>>>> >> state/timer yet in the documentation. Tracked here:
>>>>> >> https://jira.apache.org/jira/browse/BEAM-2472
>>>>> >>
>>>>> >> > I've been going over this topic a bit. I'll add the
>>>>> snippets next week, if that's fine by y'all.
>>>>> >>
>>>>> >> That would be great. The blog posts are a great way to get
>>>>> started with
>>>>> >> state/timers.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Max
>>>>> >>
>>>>> >> On 11.04.19 20:21, Pablo Estrada wrote:
>>>>> >> > I've been going over this topic a bit. I'll add the
>>>>> snippets next week,
>>>>> >> > if that's fine by y'all.
>>>>> >> > Best
>>>>> >> > -P.
>>>>> >> >
>>>>> >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw
>>>>> <robertwb@google.com <ma...@google.com>
>>>>> >> > <mailto:robertwb@google.com <ma...@google.com>>>
>>>>> wrote:
>>>>> >> >
>>>>> >> > That's a great idea! It would probably be pretty easy
>>>>> to add the
>>>>> >> > corresponding code snippets to the docs as well.
>>>>> >> >
>>>>> >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels
>>>>> <mxm@apache.org <ma...@apache.org>
>>>>> >> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
>>>>> >> > >
>>>>> >> > > Hi everyone,
>>>>> >> > >
>>>>> >> > > The Python SDK still lacks documentation on state
>>>>> and timers.
>>>>> >> > >
>>>>> >> > > As a first step, what do you think about updating
>>>>> these two blog
>>>>> >> > posts
>>>>> >> > > with the corresponding Python code?
>>>>> >> > >
>>>>> >> > >
>>>>> https://beam.apache.org/blog/2017/02/13/stateful-processing.html
>>>>> >> > >
>>>>> https://beam.apache.org/blog/2017/08/28/timely-processing.html
>>>>> >> > >
>>>>> >> > > Thanks,
>>>>> >> > > Max
>>>>> >> >
>>>>>
Re: [docs] Python State & Timers
Posted by Robert Bradshaw <ro...@google.com>.
https://github.com/apache/beam/pull/8402
On Thu, Apr 25, 2019 at 4:26 PM Robert Bradshaw <ro...@google.com> wrote:
>
> Oh, this is for the indexing example.
>
> I actually think using CombiningState is more cleaner than ValueState.
>
> https://github.com/apache/beam/blob/release-2.12.0/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py#L262
>
> (The fact that one must specify the accumulator coder is, however,
> unfortunate. We should probably infer that if we can.)
>
> On Thu, Apr 25, 2019 at 4:19 PM Robert Bradshaw <ro...@google.com> wrote:
> >
> > The desire was to avoid the implicit disallowed combination wart in
> > Python (until we could make sense of it), and also ValueState could be
> > surprising with respect to older values overwriting newer ones. What
> > was the specific example that was less natural?
> >
> > On Thu, Apr 25, 2019 at 3:01 PM Maximilian Michels <mx...@apache.org> wrote:
> > >
> > > @Pablo: Thanks for following up with the PR! :)
> > >
> > > @Brian: I was wondering about this as well. It makes the Python state
> > > code a bit unnatural. I'd suggest to add a ValueState wrapper around
> > > ListState/CombiningState.
> > >
> > > @Robert: Like Reuven pointed out, we can disallow ValueState for merging
> > > windows with state.
> > >
> > > @Reza: Great. Let's make sure it has Python examples out of the box.
> > > Either Pablo or me could help there.
> > >
> > > Thanks,
> > > Max
> > >
> > > On 25.04.19 04:14, Reza Ardeshir Rokni wrote:
> > > > Pablo, Kenneth and I have a new blog ready for publication which covers
> > > > how to create a "looping timer" it allows for default values to be
> > > > created in a window when no incoming elements exists. We just need to
> > > > clear a few bits before publication, but would be great to have that
> > > > also include a python example, I wrote it in java...
> > > >
> > > > Cheers
> > > >
> > > > Reza
> > > >
> > > > On Thu, 25 Apr 2019 at 04:34, Reuven Lax <relax@google.com
> > > > <ma...@google.com>> wrote:
> > > >
> > > > Well state is still not implemented for merging windows even for
> > > > Java (though I believe the idea was to disallow ValueState there).
> > > >
> > > > On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <robertwb@google.com
> > > > <ma...@google.com>> wrote:
> > > >
> > > > It was unclear what the semantics were for ValueState for merging
> > > > windows. (It's also a bit weird as it's inherently a race condition
> > > > wrt element ordering, unlike Bag and CombineState, though you can
> > > > always implement it as a CombineState that always returns the latest
> > > > value which is a bit more explicit about the dangers here.)
> > > >
> > > > On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette
> > > > <bhulette@google.com <ma...@google.com>> wrote:
> > > > >
> > > > > That's a great idea! I thought about this too after those
> > > > posts came up on the list recently. I started to look into it,
> > > > but I noticed that there's actually no implementation of
> > > > ValueState in userstate. Is there a reason for that? I started
> > > > to work on a patch to add it but I was just curious if there was
> > > > some reason it was omitted that I should be aware of.
> > > > >
> > > > > We could certainly replicate the example without ValueState
> > > > by using BagState and clearing it before each write, but it
> > > > would be nice if we could draw a direct parallel.
> > > > >
> > > > > Brian
> > > > >
> > > > > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>> wrote:
> > > > >>
> > > > >> > It would probably be pretty easy to add the corresponding
> > > > code snippets to the docs as well.
> > > > >>
> > > > >> It's probably a bit more work because there is no section
> > > > dedicated to
> > > > >> state/timer yet in the documentation. Tracked here:
> > > > >> https://jira.apache.org/jira/browse/BEAM-2472
> > > > >>
> > > > >> > I've been going over this topic a bit. I'll add the
> > > > snippets next week, if that's fine by y'all.
> > > > >>
> > > > >> That would be great. The blog posts are a great way to get
> > > > started with
> > > > >> state/timers.
> > > > >>
> > > > >> Thanks,
> > > > >> Max
> > > > >>
> > > > >> On 11.04.19 20:21, Pablo Estrada wrote:
> > > > >> > I've been going over this topic a bit. I'll add the
> > > > snippets next week,
> > > > >> > if that's fine by y'all.
> > > > >> > Best
> > > > >> > -P.
> > > > >> >
> > > > >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw
> > > > <robertwb@google.com <ma...@google.com>
> > > > >> > <mailto:robertwb@google.com <ma...@google.com>>>
> > > > wrote:
> > > > >> >
> > > > >> > That's a great idea! It would probably be pretty easy
> > > > to add the
> > > > >> > corresponding code snippets to the docs as well.
> > > > >> >
> > > > >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > > > >> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > > > >> > >
> > > > >> > > Hi everyone,
> > > > >> > >
> > > > >> > > The Python SDK still lacks documentation on state
> > > > and timers.
> > > > >> > >
> > > > >> > > As a first step, what do you think about updating
> > > > these two blog
> > > > >> > posts
> > > > >> > > with the corresponding Python code?
> > > > >> > >
> > > > >> > >
> > > > https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> > > > >> > >
> > > > https://beam.apache.org/blog/2017/08/28/timely-processing.html
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Max
> > > > >> >
> > > >
Re: [docs] Python State & Timers
Posted by Robert Bradshaw <ro...@google.com>.
Oh, this is for the indexing example.
I actually think using CombiningState is more cleaner than ValueState.
https://github.com/apache/beam/blob/release-2.12.0/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py#L262
(The fact that one must specify the accumulator coder is, however,
unfortunate. We should probably infer that if we can.)
On Thu, Apr 25, 2019 at 4:19 PM Robert Bradshaw <ro...@google.com> wrote:
>
> The desire was to avoid the implicit disallowed combination wart in
> Python (until we could make sense of it), and also ValueState could be
> surprising with respect to older values overwriting newer ones. What
> was the specific example that was less natural?
>
> On Thu, Apr 25, 2019 at 3:01 PM Maximilian Michels <mx...@apache.org> wrote:
> >
> > @Pablo: Thanks for following up with the PR! :)
> >
> > @Brian: I was wondering about this as well. It makes the Python state
> > code a bit unnatural. I'd suggest to add a ValueState wrapper around
> > ListState/CombiningState.
> >
> > @Robert: Like Reuven pointed out, we can disallow ValueState for merging
> > windows with state.
> >
> > @Reza: Great. Let's make sure it has Python examples out of the box.
> > Either Pablo or me could help there.
> >
> > Thanks,
> > Max
> >
> > On 25.04.19 04:14, Reza Ardeshir Rokni wrote:
> > > Pablo, Kenneth and I have a new blog ready for publication which covers
> > > how to create a "looping timer" it allows for default values to be
> > > created in a window when no incoming elements exists. We just need to
> > > clear a few bits before publication, but would be great to have that
> > > also include a python example, I wrote it in java...
> > >
> > > Cheers
> > >
> > > Reza
> > >
> > > On Thu, 25 Apr 2019 at 04:34, Reuven Lax <relax@google.com
> > > <ma...@google.com>> wrote:
> > >
> > > Well state is still not implemented for merging windows even for
> > > Java (though I believe the idea was to disallow ValueState there).
> > >
> > > On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <robertwb@google.com
> > > <ma...@google.com>> wrote:
> > >
> > > It was unclear what the semantics were for ValueState for merging
> > > windows. (It's also a bit weird as it's inherently a race condition
> > > wrt element ordering, unlike Bag and CombineState, though you can
> > > always implement it as a CombineState that always returns the latest
> > > value which is a bit more explicit about the dangers here.)
> > >
> > > On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette
> > > <bhulette@google.com <ma...@google.com>> wrote:
> > > >
> > > > That's a great idea! I thought about this too after those
> > > posts came up on the list recently. I started to look into it,
> > > but I noticed that there's actually no implementation of
> > > ValueState in userstate. Is there a reason for that? I started
> > > to work on a patch to add it but I was just curious if there was
> > > some reason it was omitted that I should be aware of.
> > > >
> > > > We could certainly replicate the example without ValueState
> > > by using BagState and clearing it before each write, but it
> > > would be nice if we could draw a direct parallel.
> > > >
> > > > Brian
> > > >
> > > > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>> wrote:
> > > >>
> > > >> > It would probably be pretty easy to add the corresponding
> > > code snippets to the docs as well.
> > > >>
> > > >> It's probably a bit more work because there is no section
> > > dedicated to
> > > >> state/timer yet in the documentation. Tracked here:
> > > >> https://jira.apache.org/jira/browse/BEAM-2472
> > > >>
> > > >> > I've been going over this topic a bit. I'll add the
> > > snippets next week, if that's fine by y'all.
> > > >>
> > > >> That would be great. The blog posts are a great way to get
> > > started with
> > > >> state/timers.
> > > >>
> > > >> Thanks,
> > > >> Max
> > > >>
> > > >> On 11.04.19 20:21, Pablo Estrada wrote:
> > > >> > I've been going over this topic a bit. I'll add the
> > > snippets next week,
> > > >> > if that's fine by y'all.
> > > >> > Best
> > > >> > -P.
> > > >> >
> > > >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw
> > > <robertwb@google.com <ma...@google.com>
> > > >> > <mailto:robertwb@google.com <ma...@google.com>>>
> > > wrote:
> > > >> >
> > > >> > That's a great idea! It would probably be pretty easy
> > > to add the
> > > >> > corresponding code snippets to the docs as well.
> > > >> >
> > > >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > > >> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > > >> > >
> > > >> > > Hi everyone,
> > > >> > >
> > > >> > > The Python SDK still lacks documentation on state
> > > and timers.
> > > >> > >
> > > >> > > As a first step, what do you think about updating
> > > these two blog
> > > >> > posts
> > > >> > > with the corresponding Python code?
> > > >> > >
> > > >> > >
> > > https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> > > >> > >
> > > https://beam.apache.org/blog/2017/08/28/timely-processing.html
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Max
> > > >> >
> > >
Re: [docs] Python State & Timers
Posted by Robert Bradshaw <ro...@google.com>.
The desire was to avoid the implicit disallowed combination wart in
Python (until we could make sense of it), and also ValueState could be
surprising with respect to older values overwriting newer ones. What
was the specific example that was less natural?
On Thu, Apr 25, 2019 at 3:01 PM Maximilian Michels <mx...@apache.org> wrote:
>
> @Pablo: Thanks for following up with the PR! :)
>
> @Brian: I was wondering about this as well. It makes the Python state
> code a bit unnatural. I'd suggest to add a ValueState wrapper around
> ListState/CombiningState.
>
> @Robert: Like Reuven pointed out, we can disallow ValueState for merging
> windows with state.
>
> @Reza: Great. Let's make sure it has Python examples out of the box.
> Either Pablo or me could help there.
>
> Thanks,
> Max
>
> On 25.04.19 04:14, Reza Ardeshir Rokni wrote:
> > Pablo, Kenneth and I have a new blog ready for publication which covers
> > how to create a "looping timer" it allows for default values to be
> > created in a window when no incoming elements exists. We just need to
> > clear a few bits before publication, but would be great to have that
> > also include a python example, I wrote it in java...
> >
> > Cheers
> >
> > Reza
> >
> > On Thu, 25 Apr 2019 at 04:34, Reuven Lax <relax@google.com
> > <ma...@google.com>> wrote:
> >
> > Well state is still not implemented for merging windows even for
> > Java (though I believe the idea was to disallow ValueState there).
> >
> > On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <robertwb@google.com
> > <ma...@google.com>> wrote:
> >
> > It was unclear what the semantics were for ValueState for merging
> > windows. (It's also a bit weird as it's inherently a race condition
> > wrt element ordering, unlike Bag and CombineState, though you can
> > always implement it as a CombineState that always returns the latest
> > value which is a bit more explicit about the dangers here.)
> >
> > On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette
> > <bhulette@google.com <ma...@google.com>> wrote:
> > >
> > > That's a great idea! I thought about this too after those
> > posts came up on the list recently. I started to look into it,
> > but I noticed that there's actually no implementation of
> > ValueState in userstate. Is there a reason for that? I started
> > to work on a patch to add it but I was just curious if there was
> > some reason it was omitted that I should be aware of.
> > >
> > > We could certainly replicate the example without ValueState
> > by using BagState and clearing it before each write, but it
> > would be nice if we could draw a direct parallel.
> > >
> > > Brian
> > >
> > > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>> wrote:
> > >>
> > >> > It would probably be pretty easy to add the corresponding
> > code snippets to the docs as well.
> > >>
> > >> It's probably a bit more work because there is no section
> > dedicated to
> > >> state/timer yet in the documentation. Tracked here:
> > >> https://jira.apache.org/jira/browse/BEAM-2472
> > >>
> > >> > I've been going over this topic a bit. I'll add the
> > snippets next week, if that's fine by y'all.
> > >>
> > >> That would be great. The blog posts are a great way to get
> > started with
> > >> state/timers.
> > >>
> > >> Thanks,
> > >> Max
> > >>
> > >> On 11.04.19 20:21, Pablo Estrada wrote:
> > >> > I've been going over this topic a bit. I'll add the
> > snippets next week,
> > >> > if that's fine by y'all.
> > >> > Best
> > >> > -P.
> > >> >
> > >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw
> > <robertwb@google.com <ma...@google.com>
> > >> > <mailto:robertwb@google.com <ma...@google.com>>>
> > wrote:
> > >> >
> > >> > That's a great idea! It would probably be pretty easy
> > to add the
> > >> > corresponding code snippets to the docs as well.
> > >> >
> > >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > >> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >> > >
> > >> > > Hi everyone,
> > >> > >
> > >> > > The Python SDK still lacks documentation on state
> > and timers.
> > >> > >
> > >> > > As a first step, what do you think about updating
> > these two blog
> > >> > posts
> > >> > > with the corresponding Python code?
> > >> > >
> > >> > >
> > https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> > >> > >
> > https://beam.apache.org/blog/2017/08/28/timely-processing.html
> > >> > >
> > >> > > Thanks,
> > >> > > Max
> > >> >
> >
Re: [docs] Python State & Timers
Posted by Maximilian Michels <mx...@apache.org>.
@Pablo: Thanks for following up with the PR! :)
@Brian: I was wondering about this as well. It makes the Python state
code a bit unnatural. I'd suggest to add a ValueState wrapper around
ListState/CombiningState.
@Robert: Like Reuven pointed out, we can disallow ValueState for merging
windows with state.
@Reza: Great. Let's make sure it has Python examples out of the box.
Either Pablo or me could help there.
Thanks,
Max
On 25.04.19 04:14, Reza Ardeshir Rokni wrote:
> Pablo, Kenneth and I have a new blog ready for publication which covers
> how to create a "looping timer" it allows for default values to be
> created in a window when no incoming elements exists. We just need to
> clear a few bits before publication, but would be great to have that
> also include a python example, I wrote it in java...
>
> Cheers
>
> Reza
>
> On Thu, 25 Apr 2019 at 04:34, Reuven Lax <relax@google.com
> <ma...@google.com>> wrote:
>
> Well state is still not implemented for merging windows even for
> Java (though I believe the idea was to disallow ValueState there).
>
> On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <robertwb@google.com
> <ma...@google.com>> wrote:
>
> It was unclear what the semantics were for ValueState for merging
> windows. (It's also a bit weird as it's inherently a race condition
> wrt element ordering, unlike Bag and CombineState, though you can
> always implement it as a CombineState that always returns the latest
> value which is a bit more explicit about the dangers here.)
>
> On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette
> <bhulette@google.com <ma...@google.com>> wrote:
> >
> > That's a great idea! I thought about this too after those
> posts came up on the list recently. I started to look into it,
> but I noticed that there's actually no implementation of
> ValueState in userstate. Is there a reason for that? I started
> to work on a patch to add it but I was just curious if there was
> some reason it was omitted that I should be aware of.
> >
> > We could certainly replicate the example without ValueState
> by using BagState and clearing it before each write, but it
> would be nice if we could draw a direct parallel.
> >
> > Brian
> >
> > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>> wrote:
> >>
> >> > It would probably be pretty easy to add the corresponding
> code snippets to the docs as well.
> >>
> >> It's probably a bit more work because there is no section
> dedicated to
> >> state/timer yet in the documentation. Tracked here:
> >> https://jira.apache.org/jira/browse/BEAM-2472
> >>
> >> > I've been going over this topic a bit. I'll add the
> snippets next week, if that's fine by y'all.
> >>
> >> That would be great. The blog posts are a great way to get
> started with
> >> state/timers.
> >>
> >> Thanks,
> >> Max
> >>
> >> On 11.04.19 20:21, Pablo Estrada wrote:
> >> > I've been going over this topic a bit. I'll add the
> snippets next week,
> >> > if that's fine by y'all.
> >> > Best
> >> > -P.
> >> >
> >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw
> <robertwb@google.com <ma...@google.com>
> >> > <mailto:robertwb@google.com <ma...@google.com>>>
> wrote:
> >> >
> >> > That's a great idea! It would probably be pretty easy
> to add the
> >> > corresponding code snippets to the docs as well.
> >> >
> >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> >> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >> > >
> >> > > Hi everyone,
> >> > >
> >> > > The Python SDK still lacks documentation on state
> and timers.
> >> > >
> >> > > As a first step, what do you think about updating
> these two blog
> >> > posts
> >> > > with the corresponding Python code?
> >> > >
> >> > >
> https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> >> > >
> https://beam.apache.org/blog/2017/08/28/timely-processing.html
> >> > >
> >> > > Thanks,
> >> > > Max
> >> >
>
Re: [docs] Python State & Timers
Posted by Reza Ardeshir Rokni <ra...@gmail.com>.
Pablo, Kenneth and I have a new blog ready for publication which covers how
to create a "looping timer" it allows for default values to be created in a
window when no incoming elements exists. We just need to clear a few bits
before publication, but would be great to have that also include a python
example, I wrote it in java...
Cheers
Reza
On Thu, 25 Apr 2019 at 04:34, Reuven Lax <re...@google.com> wrote:
> Well state is still not implemented for merging windows even for Java
> (though I believe the idea was to disallow ValueState there).
>
> On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> It was unclear what the semantics were for ValueState for merging
>> windows. (It's also a bit weird as it's inherently a race condition
>> wrt element ordering, unlike Bag and CombineState, though you can
>> always implement it as a CombineState that always returns the latest
>> value which is a bit more explicit about the dangers here.)
>>
>> On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette <bh...@google.com>
>> wrote:
>> >
>> > That's a great idea! I thought about this too after those posts came up
>> on the list recently. I started to look into it, but I noticed that there's
>> actually no implementation of ValueState in userstate. Is there a reason
>> for that? I started to work on a patch to add it but I was just curious if
>> there was some reason it was omitted that I should be aware of.
>> >
>> > We could certainly replicate the example without ValueState by using
>> BagState and clearing it before each write, but it would be nice if we
>> could draw a direct parallel.
>> >
>> > Brian
>> >
>> > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels <mx...@apache.org>
>> wrote:
>> >>
>> >> > It would probably be pretty easy to add the corresponding code
>> snippets to the docs as well.
>> >>
>> >> It's probably a bit more work because there is no section dedicated to
>> >> state/timer yet in the documentation. Tracked here:
>> >> https://jira.apache.org/jira/browse/BEAM-2472
>> >>
>> >> > I've been going over this topic a bit. I'll add the snippets next
>> week, if that's fine by y'all.
>> >>
>> >> That would be great. The blog posts are a great way to get started with
>> >> state/timers.
>> >>
>> >> Thanks,
>> >> Max
>> >>
>> >> On 11.04.19 20:21, Pablo Estrada wrote:
>> >> > I've been going over this topic a bit. I'll add the snippets next
>> week,
>> >> > if that's fine by y'all.
>> >> > Best
>> >> > -P.
>> >> >
>> >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw <robertwb@google.com
>> >> > <ma...@google.com>> wrote:
>> >> >
>> >> > That's a great idea! It would probably be pretty easy to add the
>> >> > corresponding code snippets to the docs as well.
>> >> >
>> >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels <
>> mxm@apache.org
>> >> > <ma...@apache.org>> wrote:
>> >> > >
>> >> > > Hi everyone,
>> >> > >
>> >> > > The Python SDK still lacks documentation on state and timers.
>> >> > >
>> >> > > As a first step, what do you think about updating these two
>> blog
>> >> > posts
>> >> > > with the corresponding Python code?
>> >> > >
>> >> > >
>> https://beam.apache.org/blog/2017/02/13/stateful-processing.html
>> >> > >
>> https://beam.apache.org/blog/2017/08/28/timely-processing.html
>> >> > >
>> >> > > Thanks,
>> >> > > Max
>> >> >
>>
>
Re: [docs] Python State & Timers
Posted by Reuven Lax <re...@google.com>.
Well state is still not implemented for merging windows even for Java
(though I believe the idea was to disallow ValueState there).
On Wed, Apr 24, 2019 at 1:11 PM Robert Bradshaw <ro...@google.com> wrote:
> It was unclear what the semantics were for ValueState for merging
> windows. (It's also a bit weird as it's inherently a race condition
> wrt element ordering, unlike Bag and CombineState, though you can
> always implement it as a CombineState that always returns the latest
> value which is a bit more explicit about the dangers here.)
>
> On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette <bh...@google.com>
> wrote:
> >
> > That's a great idea! I thought about this too after those posts came up
> on the list recently. I started to look into it, but I noticed that there's
> actually no implementation of ValueState in userstate. Is there a reason
> for that? I started to work on a patch to add it but I was just curious if
> there was some reason it was omitted that I should be aware of.
> >
> > We could certainly replicate the example without ValueState by using
> BagState and clearing it before each write, but it would be nice if we
> could draw a direct parallel.
> >
> > Brian
> >
> > On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels <mx...@apache.org>
> wrote:
> >>
> >> > It would probably be pretty easy to add the corresponding code
> snippets to the docs as well.
> >>
> >> It's probably a bit more work because there is no section dedicated to
> >> state/timer yet in the documentation. Tracked here:
> >> https://jira.apache.org/jira/browse/BEAM-2472
> >>
> >> > I've been going over this topic a bit. I'll add the snippets next
> week, if that's fine by y'all.
> >>
> >> That would be great. The blog posts are a great way to get started with
> >> state/timers.
> >>
> >> Thanks,
> >> Max
> >>
> >> On 11.04.19 20:21, Pablo Estrada wrote:
> >> > I've been going over this topic a bit. I'll add the snippets next
> week,
> >> > if that's fine by y'all.
> >> > Best
> >> > -P.
> >> >
> >> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw <robertwb@google.com
> >> > <ma...@google.com>> wrote:
> >> >
> >> > That's a great idea! It would probably be pretty easy to add the
> >> > corresponding code snippets to the docs as well.
> >> >
> >> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels <
> mxm@apache.org
> >> > <ma...@apache.org>> wrote:
> >> > >
> >> > > Hi everyone,
> >> > >
> >> > > The Python SDK still lacks documentation on state and timers.
> >> > >
> >> > > As a first step, what do you think about updating these two
> blog
> >> > posts
> >> > > with the corresponding Python code?
> >> > >
> >> > >
> https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> >> > > https://beam.apache.org/blog/2017/08/28/timely-processing.html
> >> > >
> >> > > Thanks,
> >> > > Max
> >> >
>
Re: [docs] Python State & Timers
Posted by Robert Bradshaw <ro...@google.com>.
It was unclear what the semantics were for ValueState for merging
windows. (It's also a bit weird as it's inherently a race condition
wrt element ordering, unlike Bag and CombineState, though you can
always implement it as a CombineState that always returns the latest
value which is a bit more explicit about the dangers here.)
On Wed, Apr 24, 2019 at 10:08 PM Brian Hulette <bh...@google.com> wrote:
>
> That's a great idea! I thought about this too after those posts came up on the list recently. I started to look into it, but I noticed that there's actually no implementation of ValueState in userstate. Is there a reason for that? I started to work on a patch to add it but I was just curious if there was some reason it was omitted that I should be aware of.
>
> We could certainly replicate the example without ValueState by using BagState and clearing it before each write, but it would be nice if we could draw a direct parallel.
>
> Brian
>
> On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels <mx...@apache.org> wrote:
>>
>> > It would probably be pretty easy to add the corresponding code snippets to the docs as well.
>>
>> It's probably a bit more work because there is no section dedicated to
>> state/timer yet in the documentation. Tracked here:
>> https://jira.apache.org/jira/browse/BEAM-2472
>>
>> > I've been going over this topic a bit. I'll add the snippets next week, if that's fine by y'all.
>>
>> That would be great. The blog posts are a great way to get started with
>> state/timers.
>>
>> Thanks,
>> Max
>>
>> On 11.04.19 20:21, Pablo Estrada wrote:
>> > I've been going over this topic a bit. I'll add the snippets next week,
>> > if that's fine by y'all.
>> > Best
>> > -P.
>> >
>> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw <robertwb@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > That's a great idea! It would probably be pretty easy to add the
>> > corresponding code snippets to the docs as well.
>> >
>> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels <mxm@apache.org
>> > <ma...@apache.org>> wrote:
>> > >
>> > > Hi everyone,
>> > >
>> > > The Python SDK still lacks documentation on state and timers.
>> > >
>> > > As a first step, what do you think about updating these two blog
>> > posts
>> > > with the corresponding Python code?
>> > >
>> > > https://beam.apache.org/blog/2017/02/13/stateful-processing.html
>> > > https://beam.apache.org/blog/2017/08/28/timely-processing.html
>> > >
>> > > Thanks,
>> > > Max
>> >
Re: [docs] Python State & Timers
Posted by Brian Hulette <bh...@google.com>.
That's a great idea! I thought about this too after those posts came up on
the list recently. I started to look into it, but I noticed that there's
actually no implementation of ValueState in userstate. Is there a reason
for that? I started to work on a patch to add it but I was just curious if
there was some reason it was omitted that I should be aware of.
We could certainly replicate the example without ValueState by using
BagState and clearing it before each write, but it would be nice if we
could draw a direct parallel.
Brian
On Fri, Apr 12, 2019 at 7:05 AM Maximilian Michels <mx...@apache.org> wrote:
> > It would probably be pretty easy to add the corresponding code snippets
> to the docs as well.
>
> It's probably a bit more work because there is no section dedicated to
> state/timer yet in the documentation. Tracked here:
> https://jira.apache.org/jira/browse/BEAM-2472
>
> > I've been going over this topic a bit. I'll add the snippets next week,
> if that's fine by y'all.
>
> That would be great. The blog posts are a great way to get started with
> state/timers.
>
> Thanks,
> Max
>
> On 11.04.19 20:21, Pablo Estrada wrote:
> > I've been going over this topic a bit. I'll add the snippets next week,
> > if that's fine by y'all.
> > Best
> > -P.
> >
> > On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw <robertwb@google.com
> > <ma...@google.com>> wrote:
> >
> > That's a great idea! It would probably be pretty easy to add the
> > corresponding code snippets to the docs as well.
> >
> > On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> > >
> > > Hi everyone,
> > >
> > > The Python SDK still lacks documentation on state and timers.
> > >
> > > As a first step, what do you think about updating these two blog
> > posts
> > > with the corresponding Python code?
> > >
> > > https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> > > https://beam.apache.org/blog/2017/08/28/timely-processing.html
> > >
> > > Thanks,
> > > Max
> >
>
Re: [docs] Python State & Timers
Posted by Maximilian Michels <mx...@apache.org>.
> It would probably be pretty easy to add the corresponding code snippets to the docs as well.
It's probably a bit more work because there is no section dedicated to
state/timer yet in the documentation. Tracked here:
https://jira.apache.org/jira/browse/BEAM-2472
> I've been going over this topic a bit. I'll add the snippets next week, if that's fine by y'all.
That would be great. The blog posts are a great way to get started with
state/timers.
Thanks,
Max
On 11.04.19 20:21, Pablo Estrada wrote:
> I've been going over this topic a bit. I'll add the snippets next week,
> if that's fine by y'all.
> Best
> -P.
>
> On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw <robertwb@google.com
> <ma...@google.com>> wrote:
>
> That's a great idea! It would probably be pretty easy to add the
> corresponding code snippets to the docs as well.
>
> On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
> >
> > Hi everyone,
> >
> > The Python SDK still lacks documentation on state and timers.
> >
> > As a first step, what do you think about updating these two blog
> posts
> > with the corresponding Python code?
> >
> > https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> > https://beam.apache.org/blog/2017/08/28/timely-processing.html
> >
> > Thanks,
> > Max
>
Re: [docs] Python State & Timers
Posted by Pablo Estrada <pa...@google.com>.
I've been going over this topic a bit. I'll add the snippets next week, if
that's fine by y'all.
Best
-P.
On Thu, Apr 11, 2019 at 5:27 AM Robert Bradshaw <ro...@google.com> wrote:
> That's a great idea! It would probably be pretty easy to add the
> corresponding code snippets to the docs as well.
>
> On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels <mx...@apache.org> wrote:
> >
> > Hi everyone,
> >
> > The Python SDK still lacks documentation on state and timers.
> >
> > As a first step, what do you think about updating these two blog posts
> > with the corresponding Python code?
> >
> > https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> > https://beam.apache.org/blog/2017/08/28/timely-processing.html
> >
> > Thanks,
> > Max
>
Re: [docs] Python State & Timers
Posted by Robert Bradshaw <ro...@google.com>.
That's a great idea! It would probably be pretty easy to add the
corresponding code snippets to the docs as well.
On Thu, Apr 11, 2019 at 2:00 PM Maximilian Michels <mx...@apache.org> wrote:
>
> Hi everyone,
>
> The Python SDK still lacks documentation on state and timers.
>
> As a first step, what do you think about updating these two blog posts
> with the corresponding Python code?
>
> https://beam.apache.org/blog/2017/02/13/stateful-processing.html
> https://beam.apache.org/blog/2017/08/28/timely-processing.html
>
> Thanks,
> Max