You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Matt <dr...@gmail.com> on 2016/12/12 13:53:34 UTC
Incremental aggregations - Example not working
Hi,
I'm following the documentation [1] of window functions with incremental
aggregations, but I'm getting an "input mismatch" error.
The code [2] is almost identical to the one in the documentation, at the
bottom you can find the exact error.
What am I missing? Can you provide a working example of a fold function
with both a FoldFunction and a WindowFunction?
Regards,
Matt
[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/windows.html#windowfunction-with-incremental-aggregation
[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
Re: Incremental aggregations - Example not working
Posted by Matt <dr...@gmail.com>.
Err, I meant if I'm not wrong *
On Mon, Dec 12, 2016 at 2:02 PM, Matt <dr...@gmail.com> wrote:
> I just checked with version 1.1.3 and it works fine, the problem is that
> in that version we can't use Kafka 0.10 if I'm not work. Thank you for the
> workaround!
>
> Best,
> Matt
>
> On Mon, Dec 12, 2016 at 1:52 PM, Yassine MARZOUGUI <
> y.marzougui@mindlytix.com> wrote:
>
>> Yes, it was suppoed to work. I looked into this, and as Chesnay said,
>> this is a bug in the fold function. I opened an issue in JIRA :
>> https://issues.apache.org/jira/browse/FLINK-5320, and will fix it very
>> soon, thank you for reporting it.
>> In the mean time you can workaround the problem by specifying the
>> TypeInformation along with the fold function as follows : fold(ACC,
>> FoldFunction, WindowFunction, foldAccumulatorType, resultType). In the
>> example, the foldAccumulatorType is new TupleTypeInfo<Tuple3<String,
>> Long, Integer>>(), and the resultType is also new
>> TupleTypeInfo<Tuple3<String, Long, Integer>>().
>>
>> Best,
>> Yassine
>>
>> 2016-12-12 16:38 GMT+01:00 Matt <dr...@gmail.com>:
>>
>>> I'm using 1.2-SNAPSHOT, should it work in that version?
>>>
>>> On Mon, Dec 12, 2016 at 12:18 PM, Yassine MARZOUGUI <
>>> y.marzougui@mindlytix.com> wrote:
>>>
>>>> Hi Matt,
>>>>
>>>> What version of Flink are you using?
>>>> The incremental agregation with fold(ACC, FoldFunction, WindowFunction)
>>>> in a new change that will be part of Flink 1.2, for Flink 1.1 the correct
>>>> way to perform incrementation aggregations is : apply(ACC,
>>>> FoldFunction, WindowFunction) (see the docs for 1.1 [1])
>>>>
>>>> [1] : https://ci.apache.org/projects/flink/flink-docs-release-1.
>>>> 1/apis/streaming/windows.html#windowfunction-with-incrementa
>>>> l-aggregation
>>>>
>>>> Best,
>>>> Yassine
>>>>
>>>> 2016-12-12 15:37 GMT+01:00 Chesnay Schepler <ch...@apache.org>:
>>>>
>>>>> Hello Matt,
>>>>>
>>>>> This looks like a bug in the fold() function to me.
>>>>>
>>>>> I'm adding Timo to the discussion, he can probably shed some light on
>>>>> this.
>>>>>
>>>>> Regards,
>>>>> Chesnay
>>>>>
>>>>>
>>>>> On 12.12.2016 15:13, Matt wrote:
>>>>>
>>>>> In case this is important, if I remove the WindowFunction, and only
>>>>> use the FoldFunction it works fine.
>>>>>
>>>>> I don't see what is wrong...
>>>>>
>>>>> On Mon, Dec 12, 2016 at 10:53 AM, Matt <dr...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm following the documentation [1] of window functions with
>>>>>> incremental aggregations, but I'm getting an "input mismatch" error.
>>>>>>
>>>>>> The code [2] is almost identical to the one in the documentation, at
>>>>>> the bottom you can find the exact error.
>>>>>>
>>>>>> What am I missing? Can you provide a working example of a fold
>>>>>> function with both a FoldFunction and a WindowFunction?
>>>>>>
>>>>>> Regards,
>>>>>> Matt
>>>>>>
>>>>>> [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/w
>>>>>> indows.html#windowfunction-with-incremental-aggregation
>>>>>>
>>>>>> [2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Incremental aggregations - Example not working
Posted by Matt <dr...@gmail.com>.
I just checked with version 1.1.3 and it works fine, the problem is that in
that version we can't use Kafka 0.10 if I'm not work. Thank you for the
workaround!
Best,
Matt
On Mon, Dec 12, 2016 at 1:52 PM, Yassine MARZOUGUI <
y.marzougui@mindlytix.com> wrote:
> Yes, it was suppoed to work. I looked into this, and as Chesnay said,
> this is a bug in the fold function. I opened an issue in JIRA :
> https://issues.apache.org/jira/browse/FLINK-5320, and will fix it very
> soon, thank you for reporting it.
> In the mean time you can workaround the problem by specifying the
> TypeInformation along with the fold function as follows : fold(ACC,
> FoldFunction, WindowFunction, foldAccumulatorType, resultType). In the
> example, the foldAccumulatorType is new TupleTypeInfo<Tuple3<String,
> Long, Integer>>(), and the resultType is also new
> TupleTypeInfo<Tuple3<String, Long, Integer>>().
>
> Best,
> Yassine
>
> 2016-12-12 16:38 GMT+01:00 Matt <dr...@gmail.com>:
>
>> I'm using 1.2-SNAPSHOT, should it work in that version?
>>
>> On Mon, Dec 12, 2016 at 12:18 PM, Yassine MARZOUGUI <
>> y.marzougui@mindlytix.com> wrote:
>>
>>> Hi Matt,
>>>
>>> What version of Flink are you using?
>>> The incremental agregation with fold(ACC, FoldFunction, WindowFunction)
>>> in a new change that will be part of Flink 1.2, for Flink 1.1 the correct
>>> way to perform incrementation aggregations is : apply(ACC,
>>> FoldFunction, WindowFunction) (see the docs for 1.1 [1])
>>>
>>> [1] : https://ci.apache.org/projects/flink/flink-docs-release-1.
>>> 1/apis/streaming/windows.html#windowfunction-with-incrementa
>>> l-aggregation
>>>
>>> Best,
>>> Yassine
>>>
>>> 2016-12-12 15:37 GMT+01:00 Chesnay Schepler <ch...@apache.org>:
>>>
>>>> Hello Matt,
>>>>
>>>> This looks like a bug in the fold() function to me.
>>>>
>>>> I'm adding Timo to the discussion, he can probably shed some light on
>>>> this.
>>>>
>>>> Regards,
>>>> Chesnay
>>>>
>>>>
>>>> On 12.12.2016 15:13, Matt wrote:
>>>>
>>>> In case this is important, if I remove the WindowFunction, and only use
>>>> the FoldFunction it works fine.
>>>>
>>>> I don't see what is wrong...
>>>>
>>>> On Mon, Dec 12, 2016 at 10:53 AM, Matt <dr...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm following the documentation [1] of window functions with
>>>>> incremental aggregations, but I'm getting an "input mismatch" error.
>>>>>
>>>>> The code [2] is almost identical to the one in the documentation, at
>>>>> the bottom you can find the exact error.
>>>>>
>>>>> What am I missing? Can you provide a working example of a fold
>>>>> function with both a FoldFunction and a WindowFunction?
>>>>>
>>>>> Regards,
>>>>> Matt
>>>>>
>>>>> [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/w
>>>>> indows.html#windowfunction-with-incremental-aggregation
>>>>>
>>>>> [2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Incremental aggregations - Example not working
Posted by Yassine MARZOUGUI <y....@mindlytix.com>.
Yes, it was suppoed to work. I looked into this, and as Chesnay said, this
is a bug in the fold function. I opened an issue in JIRA :
https://issues.apache.org/jira/browse/FLINK-5320, and will fix it very
soon, thank you for reporting it.
In the mean time you can workaround the problem by specifying the
TypeInformation along with the fold function as follows : fold(ACC,
FoldFunction, WindowFunction, foldAccumulatorType, resultType). In the
example, the foldAccumulatorType is new TupleTypeInfo<Tuple3<String, Long,
Integer>>(), and the resultType is also new TupleTypeInfo<Tuple3<String,
Long, Integer>>().
Best,
Yassine
2016-12-12 16:38 GMT+01:00 Matt <dr...@gmail.com>:
> I'm using 1.2-SNAPSHOT, should it work in that version?
>
> On Mon, Dec 12, 2016 at 12:18 PM, Yassine MARZOUGUI <
> y.marzougui@mindlytix.com> wrote:
>
>> Hi Matt,
>>
>> What version of Flink are you using?
>> The incremental agregation with fold(ACC, FoldFunction, WindowFunction)
>> in a new change that will be part of Flink 1.2, for Flink 1.1 the correct
>> way to perform incrementation aggregations is : apply(ACC, FoldFunction,
>> WindowFunction) (see the docs for 1.1 [1])
>>
>> [1] : https://ci.apache.org/projects/flink/flink-docs-release-1.
>> 1/apis/streaming/windows.html#windowfunction-with-incremental-aggregation
>>
>> Best,
>> Yassine
>>
>> 2016-12-12 15:37 GMT+01:00 Chesnay Schepler <ch...@apache.org>:
>>
>>> Hello Matt,
>>>
>>> This looks like a bug in the fold() function to me.
>>>
>>> I'm adding Timo to the discussion, he can probably shed some light on
>>> this.
>>>
>>> Regards,
>>> Chesnay
>>>
>>>
>>> On 12.12.2016 15:13, Matt wrote:
>>>
>>> In case this is important, if I remove the WindowFunction, and only use
>>> the FoldFunction it works fine.
>>>
>>> I don't see what is wrong...
>>>
>>> On Mon, Dec 12, 2016 at 10:53 AM, Matt <dr...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm following the documentation [1] of window functions with
>>>> incremental aggregations, but I'm getting an "input mismatch" error.
>>>>
>>>> The code [2] is almost identical to the one in the documentation, at
>>>> the bottom you can find the exact error.
>>>>
>>>> What am I missing? Can you provide a working example of a fold function
>>>> with both a FoldFunction and a WindowFunction?
>>>>
>>>> Regards,
>>>> Matt
>>>>
>>>> [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/w
>>>> indows.html#windowfunction-with-incremental-aggregation
>>>>
>>>> [2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
>>>>
>>>
>>>
>>>
>>
>
Re: Incremental aggregations - Example not working
Posted by Matt <dr...@gmail.com>.
I'm using 1.2-SNAPSHOT, should it work in that version?
On Mon, Dec 12, 2016 at 12:18 PM, Yassine MARZOUGUI <
y.marzougui@mindlytix.com> wrote:
> Hi Matt,
>
> What version of Flink are you using?
> The incremental agregation with fold(ACC, FoldFunction, WindowFunction)
> in a new change that will be part of Flink 1.2, for Flink 1.1 the correct
> way to perform incrementation aggregations is : apply(ACC, FoldFunction,
> WindowFunction) (see the docs for 1.1 [1])
>
> [1] : https://ci.apache.org/projects/flink/flink-docs-
> release-1.1/apis/streaming/windows.html#windowfunction-
> with-incremental-aggregation
>
> Best,
> Yassine
>
> 2016-12-12 15:37 GMT+01:00 Chesnay Schepler <ch...@apache.org>:
>
>> Hello Matt,
>>
>> This looks like a bug in the fold() function to me.
>>
>> I'm adding Timo to the discussion, he can probably shed some light on
>> this.
>>
>> Regards,
>> Chesnay
>>
>>
>> On 12.12.2016 15:13, Matt wrote:
>>
>> In case this is important, if I remove the WindowFunction, and only use
>> the FoldFunction it works fine.
>>
>> I don't see what is wrong...
>>
>> On Mon, Dec 12, 2016 at 10:53 AM, Matt <dr...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm following the documentation [1] of window functions with incremental
>>> aggregations, but I'm getting an "input mismatch" error.
>>>
>>> The code [2] is almost identical to the one in the documentation, at the
>>> bottom you can find the exact error.
>>>
>>> What am I missing? Can you provide a working example of a fold function
>>> with both a FoldFunction and a WindowFunction?
>>>
>>> Regards,
>>> Matt
>>>
>>> [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/w
>>> indows.html#windowfunction-with-incremental-aggregation
>>>
>>> [2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
>>>
>>
>>
>>
>
Re: Incremental aggregations - Example not working
Posted by Yassine MARZOUGUI <y....@mindlytix.com>.
Hi Matt,
What version of Flink are you using?
The incremental agregation with fold(ACC, FoldFunction, WindowFunction) in
a new change that will be part of Flink 1.2, for Flink 1.1 the correct way
to perform incrementation aggregations is : apply(ACC, FoldFunction,
WindowFunction) (see the docs for 1.1 [1])
[1] :
https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/windows.html#windowfunction-with-incremental-aggregation
Best,
Yassine
2016-12-12 15:37 GMT+01:00 Chesnay Schepler <ch...@apache.org>:
> Hello Matt,
>
> This looks like a bug in the fold() function to me.
>
> I'm adding Timo to the discussion, he can probably shed some light on this.
>
> Regards,
> Chesnay
>
>
> On 12.12.2016 15:13, Matt wrote:
>
> In case this is important, if I remove the WindowFunction, and only use
> the FoldFunction it works fine.
>
> I don't see what is wrong...
>
> On Mon, Dec 12, 2016 at 10:53 AM, Matt <dr...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm following the documentation [1] of window functions with incremental
>> aggregations, but I'm getting an "input mismatch" error.
>>
>> The code [2] is almost identical to the one in the documentation, at the
>> bottom you can find the exact error.
>>
>> What am I missing? Can you provide a working example of a fold function
>> with both a FoldFunction and a WindowFunction?
>>
>> Regards,
>> Matt
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/
>> windows.html#windowfunction-with-incremental-aggregation
>>
>> [2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
>>
>
>
>
Re: Incremental aggregations - Example not working
Posted by Chesnay Schepler <ch...@apache.org>.
Hello Matt,
This looks like a bug in the fold() function to me.
I'm adding Timo to the discussion, he can probably shed some light on this.
Regards,
Chesnay
On 12.12.2016 15:13, Matt wrote:
> In case this is important, if I remove the WindowFunction, and only
> use the FoldFunction it works fine.
>
> I don't see what is wrong...
>
> On Mon, Dec 12, 2016 at 10:53 AM, Matt <dromitlabs@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi,
>
> I'm following the documentation [1] of window functions with
> incremental aggregations, but I'm getting an "input mismatch" error.
>
> The code [2] is almost identical to the one in the documentation,
> at the bottom you can find the exact error.
>
> What am I missing? Can you provide a working example of a fold
> function with both a FoldFunction and a WindowFunction?
>
> Regards,
> Matt
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/dev/windows.html#windowfunction-with-incremental-aggregation
> <https://ci.apache.org/projects/flink/flink-docs-master/dev/windows.html#windowfunction-with-incremental-aggregation>
>
> [2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
> <https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d>
>
>
Re: Incremental aggregations - Example not working
Posted by Matt <dr...@gmail.com>.
In case this is important, if I remove the WindowFunction, and only use the
FoldFunction it works fine.
I don't see what is wrong...
On Mon, Dec 12, 2016 at 10:53 AM, Matt <dr...@gmail.com> wrote:
> Hi,
>
> I'm following the documentation [1] of window functions with incremental
> aggregations, but I'm getting an "input mismatch" error.
>
> The code [2] is almost identical to the one in the documentation, at the
> bottom you can find the exact error.
>
> What am I missing? Can you provide a working example of a fold function
> with both a FoldFunction and a WindowFunction?
>
> Regards,
> Matt
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> master/dev/windows.html#windowfunction-with-incremental-aggregation
>
> [2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
>