You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by lec ssmi <sh...@gmail.com> on 2020/07/01 01:53:39 UTC
the group key is retracted
Hi:
When we use sql for aggregation operation, for example, the following sql
> select count( distinct name) cnt, type from table group by type
Source data can be regarded as bin log data.
If the type value of a record is updated in the database, the values
before and after the update will be divided into different partitions and
handed over to different operators for calculation. Can Retraction happen
correctly?
Re: the group key is retracted
Posted by Benchao Li <li...@apache.org>.
If you are using 1.11 new changelog format, I think it will retract old
value from old partition correctly.
If not, (I assume you are using append only changelog) I think it won't
retract old value.
lec ssmi <sh...@gmail.com> 于2020年7月1日周三 下午2:39写道：
> The old value is already counted in a partition, and when the above update
> occurs, will the count value of the old partition be subtracted by 1, and
> then added to the new partition?
>
> Benchao Li <li...@apache.org> 于2020年7月1日周三 下午1:11写道：
>
>> Hi lec ssmi,
>>
>> > If the type value of a record is updated in the database, the values
>> before and after the update will be divided into different partitions and
>> handed over to different operators for calculation.
>> I think your understanding is correct.
>>
>> > Can Retraction happen correctly?
>> I didn't get your point, can you elaborate your question a little bit?
>>
>> lec ssmi <sh...@gmail.com> 于2020年7月1日周三 上午9:54写道：
>>
>>> Hi:
>>> When we use sql for aggregation operation, for example, the following
>>> sql
>>>
>>>> select count( distinct name) cnt, type from table group by type
>>>
>>>
>>> Source data can be regarded as bin log data.
>>> If the type value of a record is updated in the database, the values
>>> before and after the update will be divided into different partitions and
>>> handed over to different operators for calculation. Can Retraction happen
>>> correctly?
>>>
>>
>>
>> --
>>
>> Best,
>> Benchao Li
>>
>
--
Best,
Benchao Li
Re: the group key is retracted
Posted by lec ssmi <sh...@gmail.com>.
The old value is already counted in a partition, and when the above update
occurs, will the count value of the old partition be subtracted by 1, and
then added to the new partition?
Benchao Li <li...@apache.org> 于2020年7月1日周三 下午1:11写道：
> Hi lec ssmi,
>
> > If the type value of a record is updated in the database, the values
> before and after the update will be divided into different partitions and
> handed over to different operators for calculation.
> I think your understanding is correct.
>
> > Can Retraction happen correctly?
> I didn't get your point, can you elaborate your question a little bit?
>
> lec ssmi <sh...@gmail.com> 于2020年7月1日周三 上午9:54写道：
>
>> Hi:
>> When we use sql for aggregation operation, for example, the following
>> sql
>>
>>> select count( distinct name) cnt, type from table group by type
>>
>>
>> Source data can be regarded as bin log data.
>> If the type value of a record is updated in the database, the values
>> before and after the update will be divided into different partitions and
>> handed over to different operators for calculation. Can Retraction happen
>> correctly?
>>
>
>
> --
>
> Best,
> Benchao Li
>
Re: the group key is retracted
Posted by Benchao Li <li...@apache.org>.
Hi lec ssmi,
> If the type value of a record is updated in the database, the values
before and after the update will be divided into different partitions and
handed over to different operators for calculation.
I think your understanding is correct.
> Can Retraction happen correctly?
I didn't get your point, can you elaborate your question a little bit?
lec ssmi <sh...@gmail.com> 于2020年7月1日周三 上午9:54写道：
> Hi:
> When we use sql for aggregation operation, for example, the following sql
>
>> select count( distinct name) cnt, type from table group by type
>
>
> Source data can be regarded as bin log data.
> If the type value of a record is updated in the database, the values
> before and after the update will be divided into different partitions and
> handed over to different operators for calculation. Can Retraction happen
> correctly?
>
--
Best,
Benchao Li
Re: Re: the group key is retracted
Posted by 刘大龙 <ld...@zju.edu.cn>.
oh, sorry, if source operator can retract old value, I think it can.
-----原始邮件-----
发件人:"lec ssmi" <sh...@gmail.com>
发送时间:2020-07-01 10:48:38 (星期三)
收件人: "刘大龙" <ld...@zju.edu.cn>, flink-user <us...@flink.apache.org>
抄送:
主题: Re: the group key is retracted
Logically, when retracting , the old value can be sent to the original partition , as long as the group by operator supports that the two values before and after the update can be sent to different partitions. But don't know whether flink sql does this.
刘大龙 <ld...@zju.edu.cn> 于2020年7月1日周三 上午10:23写道：
I think the old value will not retract, because the type value update, it will be calculate in new value, the old value will not be updated
-----原始邮件-----
发件人:"lec ssmi" <sh...@gmail.com>
发送时间:2020-07-01 09:53:39 (星期三)
收件人: flink-user <us...@flink.apache.org>
抄送:
主题: the group key is retracted
Hi:
When we use sql for aggregation operation, for example, the following sql
select count( distinct name) cnt, type from table group by type
Source data can be regarded as bin log data.
If the type value of a record is updated in the database, the values before and after the update will be divided into different partitions and handed over to different operators for calculation. Can Retraction happen correctly?
刘大龙
浙江大学 控制系 智能系统与控制研究所 工控新楼217
地址：浙江省杭州市浙大路38号浙江大学玉泉校区
Tel：18867547281
Re: the group key is retracted
Posted by lec ssmi <sh...@gmail.com>.
Logically, when retracting , the old value can be sent to the original
partition , as long as the group by operator supports that the two values
before and after the update can be sent to different partitions. But don't
know whether flink sql does this.
刘大龙 <ld...@zju.edu.cn> 于2020年7月1日周三 上午10:23写道：
> I think the old value will not retract, because the type value update, it
> will be calculate in new value, the old value will not be updated
>
>
> -----原始邮件-----
> *发件人:*"lec ssmi" <sh...@gmail.com>
> *发送时间:*2020-07-01 09:53:39 (星期三)
> *收件人:* flink-user <us...@flink.apache.org>
> *抄送:*
> *主题:* the group key is retracted
>
> Hi:
> When we use sql for aggregation operation, for example, the following
> sql
>
>> select count( distinct name) cnt, type from table group by type
>
>
> Source data can be regarded as bin log data.
> If the type value of a record is updated in the database, the values
> before and after the update will be divided into different partitions and
> handed over to different operators for calculation. Can Retraction happen
> correctly?
>
>
>
>
> ------------------------------
> 刘大龙
>
> 浙江大学 控制系 智能系统与控制研究所 工控新楼217
> 地址：浙江省杭州市浙大路38号浙江大学玉泉校区
> Tel：18867547281
>