You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Chirag Dewan <ch...@yahoo.in> on 2018/03/12 09:20:15 UTC
Re: Partial aggregation result sink
Hi LiYue,
This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows
|
|
| |
Apache Flink 1.5-SNAPSHOT Documentation: Windows
|
|
|
So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:
@Override public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception { // schedule next timer ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L); return TriggerResult.FIRE; }
I hope it helps.
Chirag
On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <li...@gmail.com> wrote:
Hi,team
I’m working on a event-time based aggregation application with flink SQL. Is there any way to keep sinking partial aggregation result BEFORE time window closed?
For example, My SQL:
select …
from my_table
GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;
Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.
Thanks!
LiYue
Re: Partial aggregation result sink
Posted by Fabian Hueske <fh...@apache.org>.
Hi,
Yes there are plans to support early results and control the result latency
/ result completeness trade off.
However, I cannot say when these features will be available.
Best, Fabian
2018-03-19 8:14 GMT+01:00 LiYue <li...@gmail.com>:
> Hi ,
>
> Is there any plan to adding these features to flink SQL ?
>
> Thanks
> LiYue
> tig.jd.com
>
>
>
> 在 2018年3月14日，上午7:48，Fabian Hueske <fh...@apache.org> 写道：
>
> Hi,
>
> Chesnay is right.
> SQL and Table API do not support early window results and no allowed
> lateness to update results with late arriving data.
> If you need such features, you should use the DataStream API.
>
> Best, Fabian
>
>
> 2018-03-13 12:10 GMT+01:00 Chesnay Schepler <ch...@apache.org>:
>
>> I don't think you can specify custom triggers when using purer SQL, but
>> maybe Fabian or Timo know a SQL way of implementing your goal.
>>
>>
>> On 12.03.2018 13:16, 李玥 wrote:
>>
>> Hi Chirag,
>> Thank for your reply!
>> I found a provided ContinuousEventTimeTrigger should be worked in my
>> situation.
>> Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’,
>> But how to apply the trigger to a pure Flink SQL Application ?
>>
>>
>>
>>
>>
>>
>> 在 2018年3月12日，下午5:20，Chirag Dewan <ch...@yahoo.in> 写道：
>>
>> Hi LiYue,
>>
>> This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows
>> <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>>
>>
>> Apache Flink 1.5-SNAPSHOT Documentation: Windows
>> <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>>
>>
>>
>> So basically you need to register a processing time trigger at every 10
>> minutes and on callback, you can FIRE the window result like this:
>>
>> @Override
>> public TriggerResult onProcessingTime(long time, TimeWindow window,
>> TriggerContext ctx) throws Exception {
>> // schedule next timer
>> ctx.registerProcessingTimeTimer(System.currentTimeMillis() +
>> 1000L);
>> return TriggerResult.FIRE;
>> }
>>
>>
>> I hope it helps.
>>
>> Chirag
>>
>> On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <li...@gmail.com>
>> wrote:
>>
>>
>> Hi,team
>> I’m working on a event-time based aggregation application with flink
>> SQL. Is there any way to keep sinking partial aggregation result BEFORE
>> time window closed?
>> For example, My SQL:
>> select …
>> from my_table
>> GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;
>> Usually, Flink sink agg result after time-window closed, Is there any way
>> to keep sinking TODAY’s partial aggregation result every 10 miniutes so we
>> can see today’s performance on my chart.
>>
>> Thanks!
>> LiYue
>>
>>
>>
>>
>
>
Re: Partial aggregation result sink
Posted by LiYue <li...@gmail.com>.
Hi ,
Is there any plan to adding these features to flink SQL ?
Thanks
LiYue
tig.jd.com
> 在 2018年3月14日，上午7:48，Fabian Hueske <fh...@apache.org> 写道：
>
> Hi,
>
> Chesnay is right.
> SQL and Table API do not support early window results and no allowed lateness to update results with late arriving data.
> If you need such features, you should use the DataStream API.
>
> Best, Fabian
>
>
> 2018-03-13 12:10 GMT+01:00 Chesnay Schepler <chesnay@apache.org <ma...@apache.org>>:
> I don't think you can specify custom triggers when using purer SQL, but maybe Fabian or Timo know a SQL way of implementing your goal.
>
>
> On 12.03.2018 13:16, 李玥 wrote:
>> Hi Chirag,
>> Thank for your reply!
>> I found a provided ContinuousEventTimeTrigger should be worked in my situation.
>> Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’, But how to apply the trigger to a pure Flink SQL Application ?
>>
>>
>>
>>
>>
>>
>>> 在 2018年3月12日，下午5:20，Chirag Dewan <chirag.dewan22@yahoo.in <ma...@yahoo.in>> 写道：
>>>
>>> Hi LiYue,
>>>
>>> This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>>>
>>>
>>> Apache Flink 1.5-SNAPSHOT Documentation: Windows
>>>
>>> <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>>>
>>>
>>>
>>> So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:
>>>
>>> @Override
>>> public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {
>>> // schedule next timer
>>> ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);
>>> return TriggerResult.FIRE;
>>> }
>>>
>>>
>>> I hope it helps.
>>>
>>> Chirag
>>>
>>> On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <liyue2008@gmail.com <ma...@gmail.com>> wrote:
>>>
>>>
>>> Hi,team
>>> I’m working on a event-time based aggregation application with flink SQL. Is there any way to keep sinking partial aggregation result BEFORE time window closed?
>>> For example, My SQL:
>>> select …
>>> from my_table
>>> GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;
>>> Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.
>>>
>>> Thanks!
>>> LiYue
>>>
>>
>
>
Re: Partial aggregation result sink
Posted by Fabian Hueske <fh...@apache.org>.
Hi,
Chesnay is right.
SQL and Table API do not support early window results and no allowed
lateness to update results with late arriving data.
If you need such features, you should use the DataStream API.
Best, Fabian
2018-03-13 12:10 GMT+01:00 Chesnay Schepler <ch...@apache.org>:
> I don't think you can specify custom triggers when using purer SQL, but
> maybe Fabian or Timo know a SQL way of implementing your goal.
>
>
> On 12.03.2018 13:16, 李玥 wrote:
>
> Hi Chirag,
> Thank for your reply!
> I found a provided ContinuousEventTimeTrigger should be worked in my
> situation.
> Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’,
> But how to apply the trigger to a pure Flink SQL Application ?
>
>
>
>
>
>
> 在 2018年3月12日，下午5:20，Chirag Dewan <ch...@yahoo.in> 写道：
>
> Hi LiYue,
>
> This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows
> <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>
>
> Apache Flink 1.5-SNAPSHOT Documentation: Windows
> <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>
>
>
> So basically you need to register a processing time trigger at every 10
> minutes and on callback, you can FIRE the window result like this:
>
> @Override
> public TriggerResult onProcessingTime(long time, TimeWindow window,
> TriggerContext ctx) throws Exception {
> // schedule next timer
> ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);
> return TriggerResult.FIRE;
> }
>
>
> I hope it helps.
>
> Chirag
>
> On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <li...@gmail.com>
> wrote:
>
>
> Hi,team
> I’m working on a event-time based aggregation application with flink
> SQL. Is there any way to keep sinking partial aggregation result BEFORE
> time window closed?
> For example, My SQL:
> select …
> from my_table
> GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;
> Usually, Flink sink agg result after time-window closed, Is there any way
> to keep sinking TODAY’s partial aggregation result every 10 miniutes so we
> can see today’s performance on my chart.
>
> Thanks!
> LiYue
>
>
>
>
Re: Partial aggregation result sink
Posted by Chesnay Schepler <ch...@apache.org>.
I don't think you can specify custom triggers when using purer SQL, but
maybe Fabian or Timo know a SQL way of implementing your goal.
On 12.03.2018 13:16, 李玥 wrote:
> Hi Chirag,
> Thank for your reply!
> I found a provided ContinuousEventTimeTrigger should be worked in my
> situation.
> Most examples are based on Table API like
> ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’, But how to apply the
> trigger to a pure Flink SQL Application ?
>
>
>
>
>
>
>> 在 2018年3月12日，下午5:20，Chirag Dewan <chirag.dewan22@yahoo.in
>> <ma...@yahoo.in>> 写道：
>>
>> Hi LiYue,
>>
>> This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows
>> <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>>
>>
>>
>>
>>
>> Apache Flink 1.5-SNAPSHOT Documentation: Windows
>>
>> <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>>
>>
>>
>> So basically you need to register a processing time trigger at every
>> 10 minutes and on callback, you can FIRE the window result like this:
>>
>> @Override
>> public TriggerResult onProcessingTime(long time, TimeWindow
>> window, TriggerContext ctx) throws Exception {
>> // schedule next timer
>> ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);
>> return TriggerResult.FIRE;
>> }
>>
>>
>> I hope it helps.
>>
>> Chirag
>>
>> On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <liyue2008@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>>
>> Hi,team
>> I’m working on a event-time based aggregation application with
>> flink SQL. Is there any way to keep sinking partial aggregation
>> result BEFORE time window closed?
>> For example, My SQL:
>> select …
>> from my_table
>> GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;
>> Usually, Flink sink agg result after time-window closed, Is there any
>> way to keep sinking TODAY’s partial aggregation result every 10
>> miniutes so we can see today’s performance on my chart.
>>
>> Thanks!
>> LiYue
>>
>
Re: Partial aggregation result sink
Posted by 李玥 <li...@gmail.com>.
Hi Chirag,
Thank for your reply!
I found a provided ContinuousEventTimeTrigger should be worked in my situation.
Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’, But how to apply the trigger to a pure Flink SQL Application ?
> 在 2018年3月12日，下午5:20，Chirag Dewan <ch...@yahoo.in> 写道：
>
> Hi LiYue,
>
> This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>
> Apache Flink 1.5-SNAPSHOT Documentation: Windows
> <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#triggers>
>
>
>
> So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:
>
> @Override
> public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {
> // schedule next timer
> ctx.registerProcessingTimeTimer(System.currentTimeMillis() + 1000L);
> return TriggerResult.FIRE;
> }
>
>
> I hope it helps.
>
> Chirag
>
> On Monday, 12 March, 2018, 2:10:25 PM IST, 李玥 <li...@gmail.com> wrote:
>
>
> Hi,team
> I’m working on a event-time based aggregation application with flink SQL. Is there any way to keep sinking partial aggregation result BEFORE time window closed?
> For example, My SQL:
> select …
> from my_table
> GROUP BY TUMBLE(`timestamp`, INTERVAL '1’ DAY),other_column;
> Usually, Flink sink agg result after time-window closed, Is there any way to keep sinking TODAY’s partial aggregation result every 10 miniutes so we can see today’s performance on my chart.
>
> Thanks!
> LiYue
>