You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Robin Qiu <ro...@google.com> on 2019/11/21 12:54:56 UTC
Default values not supported in Combine.globally() if not windowed by GlobalWindows
Hi all,
When I run a Combine.globally() transform with a default value set, on a
PCollection windowed by a FixedWindow, I got this error message:
java.lang.IllegalStateException: Default values are not supported in
Combine.globally() if the input PCollection is not windowed by
GlobalWindows ...
(from here
<https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFnBase.java#L110>
)
Is this unsupported because it is not implemented yet, or is this a design
decision to not support default values with windows other than GlobalWIndow
(e.g. because the semantics is not well defined)? Where can I learn more
about this?
Best,
Robin
Re: Default values not supported in Combine.globally() if not
windowed by GlobalWindows
Posted by Kenneth Knowles <ke...@apache.org>.
I wasn't clear: this has worked for a long time. I wasn't suggesting a new
feature. I didn't look up the exact incantation.
Kenn
On Thu, Nov 21, 2019, 14:49 Robert Bradshaw <ro...@google.com> wrote:
> On Thu, Nov 21, 2019 at 2:46 PM Reuven Lax <re...@google.com> wrote:
> >
> > That is racy. It could also be that the side input hasn't been computed
> yet but will be available in a second.
>
> It's no more racy than any other side input lookup (as we still
> respect not declaring a window is empty until the watermark asserts
> so).
>
> > On Thu, Nov 21, 2019 at 2:37 PM Kenneth Knowles <ke...@apache.org> wrote:
> >>
> >> It can work in pull mode when it is a side input, via
> Combine.globally().asSingletonView(). Then the default is produced when the
> side input is requested and there is no data found.
> >>
> >> Kenn
> >>
> >> On Thu, Nov 21, 2019 at 10:32 AM Reuven Lax <re...@google.com> wrote:
> >>>
> >>> In particular, since windows can be data based (e.g. session windows)
> the set of windows is not always knowable in advance.
> >>>
> >>> On Thu, Nov 21, 2019 at 10:29 AM Robert Bradshaw <ro...@google.com>
> wrote:
> >>>>
> >>>> The semantics are a bit undefined--the sane extension of the model to
> >>>> support this is that the default value would show up in every window
> >>>> (otherwise which window would the default value belong to), but we
> >>>> don't have support for enumerating windows (or truncating infinite
> >>>> collections in batch mode).
> >>>>
> >>>> On Thu, Nov 21, 2019 at 4:55 AM Robin Qiu <ro...@google.com> wrote:
> >>>> >
> >>>> > Hi all,
> >>>> >
> >>>> > When I run a Combine.globally() transform with a default value set,
> on a PCollection windowed by a FixedWindow, I got this error message:
> >>>> >
> >>>> > java.lang.IllegalStateException: Default values are not supported
> in Combine.globally() if the input PCollection is not windowed by
> GlobalWindows ...
> >>>> > (from here)
> >>>> >
> >>>> > Is this unsupported because it is not implemented yet, or is this a
> design decision to not support default values with windows other than
> GlobalWIndow (e.g. because the semantics is not well defined)? Where can I
> learn more about this?
> >>>> >
> >>>> > Best,
> >>>> > Robin
>
Re: Default values not supported in Combine.globally() if not
windowed by GlobalWindows
Posted by Robert Bradshaw <ro...@google.com>.
On Thu, Nov 21, 2019 at 2:46 PM Reuven Lax <re...@google.com> wrote:
>
> That is racy. It could also be that the side input hasn't been computed yet but will be available in a second.
It's no more racy than any other side input lookup (as we still
respect not declaring a window is empty until the watermark asserts
so).
> On Thu, Nov 21, 2019 at 2:37 PM Kenneth Knowles <ke...@apache.org> wrote:
>>
>> It can work in pull mode when it is a side input, via Combine.globally().asSingletonView(). Then the default is produced when the side input is requested and there is no data found.
>>
>> Kenn
>>
>> On Thu, Nov 21, 2019 at 10:32 AM Reuven Lax <re...@google.com> wrote:
>>>
>>> In particular, since windows can be data based (e.g. session windows) the set of windows is not always knowable in advance.
>>>
>>> On Thu, Nov 21, 2019 at 10:29 AM Robert Bradshaw <ro...@google.com> wrote:
>>>>
>>>> The semantics are a bit undefined--the sane extension of the model to
>>>> support this is that the default value would show up in every window
>>>> (otherwise which window would the default value belong to), but we
>>>> don't have support for enumerating windows (or truncating infinite
>>>> collections in batch mode).
>>>>
>>>> On Thu, Nov 21, 2019 at 4:55 AM Robin Qiu <ro...@google.com> wrote:
>>>> >
>>>> > Hi all,
>>>> >
>>>> > When I run a Combine.globally() transform with a default value set, on a PCollection windowed by a FixedWindow, I got this error message:
>>>> >
>>>> > java.lang.IllegalStateException: Default values are not supported in Combine.globally() if the input PCollection is not windowed by GlobalWindows ...
>>>> > (from here)
>>>> >
>>>> > Is this unsupported because it is not implemented yet, or is this a design decision to not support default values with windows other than GlobalWIndow (e.g. because the semantics is not well defined)? Where can I learn more about this?
>>>> >
>>>> > Best,
>>>> > Robin
Re: Default values not supported in Combine.globally() if not
windowed by GlobalWindows
Posted by Reuven Lax <re...@google.com>.
That is racy. It could also be that the side input hasn't been computed yet
but will be available in a second.
On Thu, Nov 21, 2019 at 2:37 PM Kenneth Knowles <ke...@apache.org> wrote:
> It can work in pull mode when it is a side input, via
> Combine.globally().asSingletonView(). Then the default is produced when the
> side input is requested and there is no data found.
>
> Kenn
>
> On Thu, Nov 21, 2019 at 10:32 AM Reuven Lax <re...@google.com> wrote:
>
>> In particular, since windows can be data based (e.g. session windows) the
>> set of windows is not always knowable in advance.
>>
>> On Thu, Nov 21, 2019 at 10:29 AM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> The semantics are a bit undefined--the sane extension of the model to
>>> support this is that the default value would show up in every window
>>> (otherwise which window would the default value belong to), but we
>>> don't have support for enumerating windows (or truncating infinite
>>> collections in batch mode).
>>>
>>> On Thu, Nov 21, 2019 at 4:55 AM Robin Qiu <ro...@google.com> wrote:
>>> >
>>> > Hi all,
>>> >
>>> > When I run a Combine.globally() transform with a default value set, on
>>> a PCollection windowed by a FixedWindow, I got this error message:
>>> >
>>> > java.lang.IllegalStateException: Default values are not supported in
>>> Combine.globally() if the input PCollection is not windowed by
>>> GlobalWindows ...
>>> > (from here)
>>> >
>>> > Is this unsupported because it is not implemented yet, or is this a
>>> design decision to not support default values with windows other than
>>> GlobalWIndow (e.g. because the semantics is not well defined)? Where can I
>>> learn more about this?
>>> >
>>> > Best,
>>> > Robin
>>>
>>
Re: Default values not supported in Combine.globally() if not
windowed by GlobalWindows
Posted by Kenneth Knowles <ke...@apache.org>.
It can work in pull mode when it is a side input, via
Combine.globally().asSingletonView(). Then the default is produced when the
side input is requested and there is no data found.
Kenn
On Thu, Nov 21, 2019 at 10:32 AM Reuven Lax <re...@google.com> wrote:
> In particular, since windows can be data based (e.g. session windows) the
> set of windows is not always knowable in advance.
>
> On Thu, Nov 21, 2019 at 10:29 AM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> The semantics are a bit undefined--the sane extension of the model to
>> support this is that the default value would show up in every window
>> (otherwise which window would the default value belong to), but we
>> don't have support for enumerating windows (or truncating infinite
>> collections in batch mode).
>>
>> On Thu, Nov 21, 2019 at 4:55 AM Robin Qiu <ro...@google.com> wrote:
>> >
>> > Hi all,
>> >
>> > When I run a Combine.globally() transform with a default value set, on
>> a PCollection windowed by a FixedWindow, I got this error message:
>> >
>> > java.lang.IllegalStateException: Default values are not supported in
>> Combine.globally() if the input PCollection is not windowed by
>> GlobalWindows ...
>> > (from here)
>> >
>> > Is this unsupported because it is not implemented yet, or is this a
>> design decision to not support default values with windows other than
>> GlobalWIndow (e.g. because the semantics is not well defined)? Where can I
>> learn more about this?
>> >
>> > Best,
>> > Robin
>>
>
Re: Default values not supported in Combine.globally() if not
windowed by GlobalWindows
Posted by Robert Bradshaw <ro...@google.com>.
On Thu, Nov 21, 2019 at 10:32 AM Reuven Lax <re...@google.com> wrote:
>
> In particular, since windows can be data based (e.g. session windows) the set of windows is not always knowable in advance.
It's [IntervalWindow(start, end) for start in range(MIN_TIMESTAMP,
MAX_TIMESTAMP) for end in range(MIN_TIMESTAMP, MAX_TIMESTAMP) if start
< end], though admittedly most have no data :) The statement is,
however, true in general.
> On Thu, Nov 21, 2019 at 10:29 AM Robert Bradshaw <ro...@google.com> wrote:
>>
>> The semantics are a bit undefined--the sane extension of the model to
>> support this is that the default value would show up in every window
>> (otherwise which window would the default value belong to), but we
>> don't have support for enumerating windows (or truncating infinite
>> collections in batch mode).
>>
>> On Thu, Nov 21, 2019 at 4:55 AM Robin Qiu <ro...@google.com> wrote:
>> >
>> > Hi all,
>> >
>> > When I run a Combine.globally() transform with a default value set, on a PCollection windowed by a FixedWindow, I got this error message:
>> >
>> > java.lang.IllegalStateException: Default values are not supported in Combine.globally() if the input PCollection is not windowed by GlobalWindows ...
>> > (from here)
>> >
>> > Is this unsupported because it is not implemented yet, or is this a design decision to not support default values with windows other than GlobalWIndow (e.g. because the semantics is not well defined)? Where can I learn more about this?
>> >
>> > Best,
>> > Robin
Re: Default values not supported in Combine.globally() if not
windowed by GlobalWindows
Posted by Reuven Lax <re...@google.com>.
In particular, since windows can be data based (e.g. session windows) the
set of windows is not always knowable in advance.
On Thu, Nov 21, 2019 at 10:29 AM Robert Bradshaw <ro...@google.com>
wrote:
> The semantics are a bit undefined--the sane extension of the model to
> support this is that the default value would show up in every window
> (otherwise which window would the default value belong to), but we
> don't have support for enumerating windows (or truncating infinite
> collections in batch mode).
>
> On Thu, Nov 21, 2019 at 4:55 AM Robin Qiu <ro...@google.com> wrote:
> >
> > Hi all,
> >
> > When I run a Combine.globally() transform with a default value set, on a
> PCollection windowed by a FixedWindow, I got this error message:
> >
> > java.lang.IllegalStateException: Default values are not supported in
> Combine.globally() if the input PCollection is not windowed by
> GlobalWindows ...
> > (from here)
> >
> > Is this unsupported because it is not implemented yet, or is this a
> design decision to not support default values with windows other than
> GlobalWIndow (e.g. because the semantics is not well defined)? Where can I
> learn more about this?
> >
> > Best,
> > Robin
>
Re: Default values not supported in Combine.globally() if not
windowed by GlobalWindows
Posted by Robert Bradshaw <ro...@google.com>.
The semantics are a bit undefined--the sane extension of the model to
support this is that the default value would show up in every window
(otherwise which window would the default value belong to), but we
don't have support for enumerating windows (or truncating infinite
collections in batch mode).
On Thu, Nov 21, 2019 at 4:55 AM Robin Qiu <ro...@google.com> wrote:
>
> Hi all,
>
> When I run a Combine.globally() transform with a default value set, on a PCollection windowed by a FixedWindow, I got this error message:
>
> java.lang.IllegalStateException: Default values are not supported in Combine.globally() if the input PCollection is not windowed by GlobalWindows ...
> (from here)
>
> Is this unsupported because it is not implemented yet, or is this a design decision to not support default values with windows other than GlobalWIndow (e.g. because the semantics is not well defined)? Where can I learn more about this?
>
> Best,
> Robin