You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Vilhelm von Ehrenheim <vo...@gmail.com> on 2018/02/26 10:47:23 UTC
Count on non-global windows
Hi!
I am trying to use the `Count` Combine transform on Fixed windows and get
the following error:
```
java.lang.IllegalStateException: Default values are not supported in
Combine.globally() if the output PCollection is not windowed by
GlobalWindows. Instead, use Combine.globally().withoutDefaults() to output
an empty PCollection if the input PCollection is empty, or
Combine.globally().asSingletonView() to get the default output of the
CombineFn if the input PCollection is empty.
```
I get that you need to set this on combiners for non-global windows but it
is not possible to set for the `Count.globally()` transform. I can
implement this using my own CompbineFn or `Sum` transform instead of course
but still thought it was a bit strange.
Is this a bug or why is this happening?
Regards,
Vilhelm von Ehrenheim
Re: Count on non-global windows
Posted by Kenneth Knowles <kl...@google.com>.
In fact, there is a framework in place for giving a better error message
that is just not used. Filed https://issues.apache.org/jira/browse/BEAM-3746
(maybe took longer to file than to fix, but I'm just not at an IDE right
now).
On Mon, Feb 26, 2018 at 8:26 AM, Kenneth Knowles <kl...@google.com> wrote:
> Hi Vilhelm,
>
> Sorry, this is not a good experience. You analysis is spot on, and the
> workaround you suggested is what we have done. It is a known issue that
> there is a tension between all the helper methods of Combine and the nice
> wrappers like Count. So for now it is just in the fine print of the
> documentation: https://beam.apache.org/documentation/sdks/
> javadoc/2.3.0/org/apache/beam/sdk/transforms/Count.html#globally--
>
> "Note: if the input collection uses a windowing strategy other than
> GlobalWindows, use Combine.globally(Count.<T>combineFn()).withoutDefaults()
> instead."
>
> Kenn
>
> On Mon, Feb 26, 2018 at 2:47 AM, Vilhelm von Ehrenheim <
> vonehrenheim@gmail.com> wrote:
>
>> Hi!
>> I am trying to use the `Count` Combine transform on Fixed windows and get
>> the following error:
>>
>> ```
>> java.lang.IllegalStateException: Default values are not supported in
>> Combine.globally() if the output PCollection is not windowed by
>> GlobalWindows. Instead, use Combine.globally().withoutDefaults() to
>> output an empty PCollection if the input PCollection is empty, or
>> Combine.globally().asSingletonView() to get the default output of the
>> CombineFn if the input PCollection is empty.
>> ```
>>
>> I get that you need to set this on combiners for non-global windows but
>> it is not possible to set for the `Count.globally()` transform. I can
>> implement this using my own CompbineFn or `Sum` transform instead of course
>> but still thought it was a bit strange.
>>
>> Is this a bug or why is this happening?
>>
>> Regards,
>> Vilhelm von Ehrenheim
>>
>
>
Re: Count on non-global windows
Posted by Kenneth Knowles <kl...@google.com>.
In fact, there is a framework in place for giving a better error message
that is just not used. Filed https://issues.apache.org/jira/browse/BEAM-3746
(maybe took longer to file than to fix, but I'm just not at an IDE right
now).
On Mon, Feb 26, 2018 at 8:26 AM, Kenneth Knowles <kl...@google.com> wrote:
> Hi Vilhelm,
>
> Sorry, this is not a good experience. You analysis is spot on, and the
> workaround you suggested is what we have done. It is a known issue that
> there is a tension between all the helper methods of Combine and the nice
> wrappers like Count. So for now it is just in the fine print of the
> documentation: https://beam.apache.org/documentation/sdks/
> javadoc/2.3.0/org/apache/beam/sdk/transforms/Count.html#globally--
>
> "Note: if the input collection uses a windowing strategy other than
> GlobalWindows, use Combine.globally(Count.<T>combineFn()).withoutDefaults()
> instead."
>
> Kenn
>
> On Mon, Feb 26, 2018 at 2:47 AM, Vilhelm von Ehrenheim <
> vonehrenheim@gmail.com> wrote:
>
>> Hi!
>> I am trying to use the `Count` Combine transform on Fixed windows and get
>> the following error:
>>
>> ```
>> java.lang.IllegalStateException: Default values are not supported in
>> Combine.globally() if the output PCollection is not windowed by
>> GlobalWindows. Instead, use Combine.globally().withoutDefaults() to
>> output an empty PCollection if the input PCollection is empty, or
>> Combine.globally().asSingletonView() to get the default output of the
>> CombineFn if the input PCollection is empty.
>> ```
>>
>> I get that you need to set this on combiners for non-global windows but
>> it is not possible to set for the `Count.globally()` transform. I can
>> implement this using my own CompbineFn or `Sum` transform instead of course
>> but still thought it was a bit strange.
>>
>> Is this a bug or why is this happening?
>>
>> Regards,
>> Vilhelm von Ehrenheim
>>
>
>
Re: Count on non-global windows
Posted by Kenneth Knowles <kl...@google.com>.
Hi Vilhelm,
Sorry, this is not a good experience. You analysis is spot on, and the
workaround you suggested is what we have done. It is a known issue that
there is a tension between all the helper methods of Combine and the nice
wrappers like Count. So for now it is just in the fine print of the
documentation:
https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/transforms/Count.html#globally--
"Note: if the input collection uses a windowing strategy other than
GlobalWindows, use Combine.globally(Count.<T>combineFn()).withoutDefaults()
instead."
Kenn
On Mon, Feb 26, 2018 at 2:47 AM, Vilhelm von Ehrenheim <
vonehrenheim@gmail.com> wrote:
> Hi!
> I am trying to use the `Count` Combine transform on Fixed windows and get
> the following error:
>
> ```
> java.lang.IllegalStateException: Default values are not supported in
> Combine.globally() if the output PCollection is not windowed by
> GlobalWindows. Instead, use Combine.globally().withoutDefaults() to
> output an empty PCollection if the input PCollection is empty, or
> Combine.globally().asSingletonView() to get the default output of the
> CombineFn if the input PCollection is empty.
> ```
>
> I get that you need to set this on combiners for non-global windows but it
> is not possible to set for the `Count.globally()` transform. I can
> implement this using my own CompbineFn or `Sum` transform instead of course
> but still thought it was a bit strange.
>
> Is this a bug or why is this happening?
>
> Regards,
> Vilhelm von Ehrenheim
>