You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by antonio saldivar <an...@gmail.com> on 2018/08/16 22:23:52 UTC
processWindowFunction
Hello
I am implementing a data stream where I use sliding windows but I am stuck
because I need to set values to my object based on some if statements in my
process function and send the object to the next step but I don't want
results every time a window is creating
if anyone has a good example on this that can help me
Re: processWindowFunction
Posted by antonio saldivar <an...@gmail.com>.
Maybe the usage of that function change, now I have to use it as this [1]
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#the-keyedprocessfunction
El lun., 20 ago. 2018 a las 5:56, vino yang (<ya...@gmail.com>)
escribió:
> Hi antonio,
>
> Oh, if you can't use KeyedProcessFunction, then this would be a pity.
> Then you can use MapState, where Key is used to store the key of your
> partition.
> But I am not sure if this will achieve the effect you want.
>
> Thanks, vino.
>
> antonio saldivar <an...@gmail.com> 于2018年8月20日周一 下午4:32写道：
>
>> Hello
>>
>> Thank you for the information, for some reason this KeyedProcessFunction
>> is not found in my Flink version 1.4.2 I can only find ProcessFunction and
>> work like this
>>
>> public class TxnProcessFn extends ProcessFunction<Object,Object> {
>>
>> public void open(Configuration parameters) throws Exception {
>>
>> state1 = getRuntimeContext().getState(new ValueStateDescriptor<>(
>> "objState1", Object.class));
>>
>> state2 = getRuntimeContext().getState(new ValueStateDescriptor<>(
>> "objState2", Object.class));
>>
>> state3 = getRuntimeContext().getState(new ValueStateDescriptor<>(
>> "objState3", Object.class));
>>
>> }
>>
>> @Override
>>
>> public void processElement(
>>
>> Object obj,
>>
>> Context ctx,
>>
>> Collector<Transaction> out) throws Exception {
>>
>> // TODO Auto-generated method stub
>>
>> Object current = state.value();
>>
>> if (current == null) {
>>
>> current = new Object();
>>
>> current.id=obj.id();
>>
>>
>>
>> }
>>
>> }
>>
>> El lun., 20 ago. 2018 a las 2:24, vino yang (<ya...@gmail.com>)
>> escribió:
>>
>>> Hi antonio,
>>>
>>> First, I suggest you use KeyedProcessFunction if you have an operation
>>> similar to keyBy.
>>> The implementation is similar to the Fixed window.
>>> You can create three state collections to determine whether the time of
>>> each element belongs to a state collection.
>>> At the time of the trigger, the elements in the collection are evaluated.
>>>
>>> Thanks, vino.
>>>
>>> antonio saldivar <an...@gmail.com> 于2018年8月20日周一 上午11:54写道：
>>>
>>>> Thank you fro the references
>>>>
>>>> I have now my processFunction and getting the state but now how can i
>>>> do for the threshold times to group the elements and also as this is a
>>>> global window, how to purge because if going to keep increasing
>>>>
>>>> El dom., 19 ago. 2018 a las 8:57, vino yang (<ya...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi antonio,
>>>>>
>>>>> Regarding your scenario, I think maybe you can consider using the
>>>>> ProcessFunction (or keyed ProcessFunction) function directly on the Stream.
>>>>> [1]
>>>>> It can handle each of your elements with a Timer, and you can combine
>>>>> Flink's state API[2] to store your data.
>>>>>
>>>>> [1]:
>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations
>>>>> [2]:
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state
>>>>>
>>>>> Thanks, vino.
>>>>>
>>>>> antonio saldivar <an...@gmail.com> 于2018年8月19日周日 上午10:18写道：
>>>>>
>>>>>> hi Vino
>>>>>>
>>>>>> it is possible to use global window, then set the trigger onElement
>>>>>> comparing the element that has arrived with for example 10 mins, 20 mins
>>>>>> and 60 mins of data?
>>>>>>
>>>>>> I have rules evaluating sum of amount for 10,20 or 60 mins for the
>>>>>> same keyed element if the same id sum like $200 total within those
>>>>>> thresholds and count more or equals to 3 I need to be able to set some
>>>>>> values to the object if the object does not reach those thresholds i do not
>>>>>> set the values and keep sending the output with or without those value.
>>>>>>
>>>>>> just processing the object on the fly and send output
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> El vie., 17 ago. 2018 a las 22:14, vino yang (<ya...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi antonio,
>>>>>>>
>>>>>>> Yes, ProcessWindowFunction is a very low level window function.
>>>>>>> It allows you to access the data in the window and allows you to
>>>>>>> customize the output of the window.
>>>>>>> So if you use it, while giving you flexibility, you need to think
>>>>>>> about other things, which may require you to write more processing logic.
>>>>>>>
>>>>>>> Generally speaking, sliding windows usually have some data that is
>>>>>>> repeated, but a common mode is to apply a reduce function on it to get your
>>>>>>> calculation results.
>>>>>>> If you only send data, there will definitely be some duplication.
>>>>>>>
>>>>>>> Thanks, vino.
>>>>>>>
>>>>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 下午12:01写道：
>>>>>>>
>>>>>>>> Hi Vino
>>>>>>>> thank you for the information, actually I am using a trigger alert
>>>>>>>> and processWindowFunction to send my results, but when my window slides or
>>>>>>>> ends it sends again the objects and I an getting duplicated data
>>>>>>>>
>>>>>>>> El jue., 16 ago. 2018 a las 22:05, vino yang (<
>>>>>>>> yanghua1127@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Hi Antonio,
>>>>>>>>>
>>>>>>>>> What results do not you want to get when creating each window?
>>>>>>>>> Examples of the use of ProcessWindowFunction are included in many
>>>>>>>>> test files in Flink's project, such as SideOutputITCase.scala or
>>>>>>>>> WindowTranslationTest.scala.
>>>>>>>>>
>>>>>>>>> For more information on ProcessWindowFunction, you can refer to
>>>>>>>>> the official website.[1]
>>>>>>>>>
>>>>>>>>> [1]:
>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>>>>>>>>>
>>>>>>>>> Thanks, vino.
>>>>>>>>>
>>>>>>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
>>>>>>>>>
>>>>>>>>>> Hello
>>>>>>>>>>
>>>>>>>>>> I am implementing a data stream where I use sliding windows but I
>>>>>>>>>> am stuck because I need to set values to my object based on some if
>>>>>>>>>> statements in my process function and send the object to the next step but
>>>>>>>>>> I don't want results every time a window is creating
>>>>>>>>>>
>>>>>>>>>> if anyone has a good example on this that can help me
>>>>>>>>>>
>>>>>>>>>
Re: processWindowFunction
Posted by vino yang <ya...@gmail.com>.
Hi antonio,
Oh, if you can't use KeyedProcessFunction, then this would be a pity.
Then you can use MapState, where Key is used to store the key of your
partition.
But I am not sure if this will achieve the effect you want.
Thanks, vino.
antonio saldivar <an...@gmail.com> 于2018年8月20日周一 下午4:32写道：
> Hello
>
> Thank you for the information, for some reason this KeyedProcessFunction
> is not found in my Flink version 1.4.2 I can only find ProcessFunction and
> work like this
>
> public class TxnProcessFn extends ProcessFunction<Object,Object> {
>
> public void open(Configuration parameters) throws Exception {
>
> state1 = getRuntimeContext().getState(new ValueStateDescriptor<>(
> "objState1", Object.class));
>
> state2 = getRuntimeContext().getState(new ValueStateDescriptor<>(
> "objState2", Object.class));
>
> state3 = getRuntimeContext().getState(new ValueStateDescriptor<>(
> "objState3", Object.class));
>
> }
>
> @Override
>
> public void processElement(
>
> Object obj,
>
> Context ctx,
>
> Collector<Transaction> out) throws Exception {
>
> // TODO Auto-generated method stub
>
> Object current = state.value();
>
> if (current == null) {
>
> current = new Object();
>
> current.id=obj.id();
>
>
>
> }
>
> }
>
> El lun., 20 ago. 2018 a las 2:24, vino yang (<ya...@gmail.com>)
> escribió:
>
>> Hi antonio,
>>
>> First, I suggest you use KeyedProcessFunction if you have an operation
>> similar to keyBy.
>> The implementation is similar to the Fixed window.
>> You can create three state collections to determine whether the time of
>> each element belongs to a state collection.
>> At the time of the trigger, the elements in the collection are evaluated.
>>
>> Thanks, vino.
>>
>> antonio saldivar <an...@gmail.com> 于2018年8月20日周一 上午11:54写道：
>>
>>> Thank you fro the references
>>>
>>> I have now my processFunction and getting the state but now how can i do
>>> for the threshold times to group the elements and also as this is a global
>>> window, how to purge because if going to keep increasing
>>>
>>> El dom., 19 ago. 2018 a las 8:57, vino yang (<ya...@gmail.com>)
>>> escribió:
>>>
>>>> Hi antonio,
>>>>
>>>> Regarding your scenario, I think maybe you can consider using the
>>>> ProcessFunction (or keyed ProcessFunction) function directly on the Stream.
>>>> [1]
>>>> It can handle each of your elements with a Timer, and you can combine
>>>> Flink's state API[2] to store your data.
>>>>
>>>> [1]:
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations
>>>> [2]:
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state
>>>>
>>>> Thanks, vino.
>>>>
>>>> antonio saldivar <an...@gmail.com> 于2018年8月19日周日 上午10:18写道：
>>>>
>>>>> hi Vino
>>>>>
>>>>> it is possible to use global window, then set the trigger onElement
>>>>> comparing the element that has arrived with for example 10 mins, 20 mins
>>>>> and 60 mins of data?
>>>>>
>>>>> I have rules evaluating sum of amount for 10,20 or 60 mins for the
>>>>> same keyed element if the same id sum like $200 total within those
>>>>> thresholds and count more or equals to 3 I need to be able to set some
>>>>> values to the object if the object does not reach those thresholds i do not
>>>>> set the values and keep sending the output with or without those value.
>>>>>
>>>>> just processing the object on the fly and send output
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> El vie., 17 ago. 2018 a las 22:14, vino yang (<ya...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi antonio,
>>>>>>
>>>>>> Yes, ProcessWindowFunction is a very low level window function.
>>>>>> It allows you to access the data in the window and allows you to
>>>>>> customize the output of the window.
>>>>>> So if you use it, while giving you flexibility, you need to think
>>>>>> about other things, which may require you to write more processing logic.
>>>>>>
>>>>>> Generally speaking, sliding windows usually have some data that is
>>>>>> repeated, but a common mode is to apply a reduce function on it to get your
>>>>>> calculation results.
>>>>>> If you only send data, there will definitely be some duplication.
>>>>>>
>>>>>> Thanks, vino.
>>>>>>
>>>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 下午12:01写道：
>>>>>>
>>>>>>> Hi Vino
>>>>>>> thank you for the information, actually I am using a trigger alert
>>>>>>> and processWindowFunction to send my results, but when my window slides or
>>>>>>> ends it sends again the objects and I an getting duplicated data
>>>>>>>
>>>>>>> El jue., 16 ago. 2018 a las 22:05, vino yang (<ya...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi Antonio,
>>>>>>>>
>>>>>>>> What results do not you want to get when creating each window?
>>>>>>>> Examples of the use of ProcessWindowFunction are included in many
>>>>>>>> test files in Flink's project, such as SideOutputITCase.scala or
>>>>>>>> WindowTranslationTest.scala.
>>>>>>>>
>>>>>>>> For more information on ProcessWindowFunction, you can refer to the
>>>>>>>> official website.[1]
>>>>>>>>
>>>>>>>> [1]:
>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>>>>>>>>
>>>>>>>> Thanks, vino.
>>>>>>>>
>>>>>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
>>>>>>>>
>>>>>>>>> Hello
>>>>>>>>>
>>>>>>>>> I am implementing a data stream where I use sliding windows but I
>>>>>>>>> am stuck because I need to set values to my object based on some if
>>>>>>>>> statements in my process function and send the object to the next step but
>>>>>>>>> I don't want results every time a window is creating
>>>>>>>>>
>>>>>>>>> if anyone has a good example on this that can help me
>>>>>>>>>
>>>>>>>>
Re: processWindowFunction
Posted by antonio saldivar <an...@gmail.com>.
Hello
Thank you for the information, for some reason this KeyedProcessFunction is
not found in my Flink version 1.4.2 I can only find ProcessFunction and
work like this
public class TxnProcessFn extends ProcessFunction<Object,Object> {
public void open(Configuration parameters) throws Exception {
state1 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState1",
Object.class));
state2 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState2",
Object.class));
state3 = getRuntimeContext().getState(new ValueStateDescriptor<>("objState3",
Object.class));
}
@Override
public void processElement(
Object obj,
Context ctx,
Collector<Transaction> out) throws Exception {
// TODO Auto-generated method stub
Object current = state.value();
if (current == null) {
current = new Object();
current.id=obj.id();
}
}
El lun., 20 ago. 2018 a las 2:24, vino yang (<ya...@gmail.com>)
escribió:
> Hi antonio,
>
> First, I suggest you use KeyedProcessFunction if you have an operation
> similar to keyBy.
> The implementation is similar to the Fixed window.
> You can create three state collections to determine whether the time of
> each element belongs to a state collection.
> At the time of the trigger, the elements in the collection are evaluated.
>
> Thanks, vino.
>
> antonio saldivar <an...@gmail.com> 于2018年8月20日周一 上午11:54写道：
>
>> Thank you fro the references
>>
>> I have now my processFunction and getting the state but now how can i do
>> for the threshold times to group the elements and also as this is a global
>> window, how to purge because if going to keep increasing
>>
>> El dom., 19 ago. 2018 a las 8:57, vino yang (<ya...@gmail.com>)
>> escribió:
>>
>>> Hi antonio,
>>>
>>> Regarding your scenario, I think maybe you can consider using the
>>> ProcessFunction (or keyed ProcessFunction) function directly on the Stream.
>>> [1]
>>> It can handle each of your elements with a Timer, and you can combine
>>> Flink's state API[2] to store your data.
>>>
>>> [1]:
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations
>>> [2]:
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state
>>>
>>> Thanks, vino.
>>>
>>> antonio saldivar <an...@gmail.com> 于2018年8月19日周日 上午10:18写道：
>>>
>>>> hi Vino
>>>>
>>>> it is possible to use global window, then set the trigger onElement
>>>> comparing the element that has arrived with for example 10 mins, 20 mins
>>>> and 60 mins of data?
>>>>
>>>> I have rules evaluating sum of amount for 10,20 or 60 mins for the same
>>>> keyed element if the same id sum like $200 total within those thresholds
>>>> and count more or equals to 3 I need to be able to set some values to the
>>>> object if the object does not reach those thresholds i do not set the
>>>> values and keep sending the output with or without those value.
>>>>
>>>> just processing the object on the fly and send output
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> El vie., 17 ago. 2018 a las 22:14, vino yang (<ya...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi antonio,
>>>>>
>>>>> Yes, ProcessWindowFunction is a very low level window function.
>>>>> It allows you to access the data in the window and allows you to
>>>>> customize the output of the window.
>>>>> So if you use it, while giving you flexibility, you need to think
>>>>> about other things, which may require you to write more processing logic.
>>>>>
>>>>> Generally speaking, sliding windows usually have some data that is
>>>>> repeated, but a common mode is to apply a reduce function on it to get your
>>>>> calculation results.
>>>>> If you only send data, there will definitely be some duplication.
>>>>>
>>>>> Thanks, vino.
>>>>>
>>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 下午12:01写道：
>>>>>
>>>>>> Hi Vino
>>>>>> thank you for the information, actually I am using a trigger alert
>>>>>> and processWindowFunction to send my results, but when my window slides or
>>>>>> ends it sends again the objects and I an getting duplicated data
>>>>>>
>>>>>> El jue., 16 ago. 2018 a las 22:05, vino yang (<ya...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi Antonio,
>>>>>>>
>>>>>>> What results do not you want to get when creating each window?
>>>>>>> Examples of the use of ProcessWindowFunction are included in many
>>>>>>> test files in Flink's project, such as SideOutputITCase.scala or
>>>>>>> WindowTranslationTest.scala.
>>>>>>>
>>>>>>> For more information on ProcessWindowFunction, you can refer to the
>>>>>>> official website.[1]
>>>>>>>
>>>>>>> [1]:
>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>>>>>>>
>>>>>>> Thanks, vino.
>>>>>>>
>>>>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
>>>>>>>
>>>>>>>> Hello
>>>>>>>>
>>>>>>>> I am implementing a data stream where I use sliding windows but I
>>>>>>>> am stuck because I need to set values to my object based on some if
>>>>>>>> statements in my process function and send the object to the next step but
>>>>>>>> I don't want results every time a window is creating
>>>>>>>>
>>>>>>>> if anyone has a good example on this that can help me
>>>>>>>>
>>>>>>>
Re: processWindowFunction
Posted by vino yang <ya...@gmail.com>.
Hi antonio,
First, I suggest you use KeyedProcessFunction if you have an operation
similar to keyBy.
The implementation is similar to the Fixed window.
You can create three state collections to determine whether the time of
each element belongs to a state collection.
At the time of the trigger, the elements in the collection are evaluated.
Thanks, vino.
antonio saldivar <an...@gmail.com> 于2018年8月20日周一 上午11:54写道：
> Thank you fro the references
>
> I have now my processFunction and getting the state but now how can i do
> for the threshold times to group the elements and also as this is a global
> window, how to purge because if going to keep increasing
>
> El dom., 19 ago. 2018 a las 8:57, vino yang (<ya...@gmail.com>)
> escribió:
>
>> Hi antonio,
>>
>> Regarding your scenario, I think maybe you can consider using the
>> ProcessFunction (or keyed ProcessFunction) function directly on the Stream.
>> [1]
>> It can handle each of your elements with a Timer, and you can combine
>> Flink's state API[2] to store your data.
>>
>> [1]:
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations
>> [2]:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state
>>
>> Thanks, vino.
>>
>> antonio saldivar <an...@gmail.com> 于2018年8月19日周日 上午10:18写道：
>>
>>> hi Vino
>>>
>>> it is possible to use global window, then set the trigger onElement
>>> comparing the element that has arrived with for example 10 mins, 20 mins
>>> and 60 mins of data?
>>>
>>> I have rules evaluating sum of amount for 10,20 or 60 mins for the same
>>> keyed element if the same id sum like $200 total within those thresholds
>>> and count more or equals to 3 I need to be able to set some values to the
>>> object if the object does not reach those thresholds i do not set the
>>> values and keep sending the output with or without those value.
>>>
>>> just processing the object on the fly and send output
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> El vie., 17 ago. 2018 a las 22:14, vino yang (<ya...@gmail.com>)
>>> escribió:
>>>
>>>> Hi antonio,
>>>>
>>>> Yes, ProcessWindowFunction is a very low level window function.
>>>> It allows you to access the data in the window and allows you to
>>>> customize the output of the window.
>>>> So if you use it, while giving you flexibility, you need to think about
>>>> other things, which may require you to write more processing logic.
>>>>
>>>> Generally speaking, sliding windows usually have some data that is
>>>> repeated, but a common mode is to apply a reduce function on it to get your
>>>> calculation results.
>>>> If you only send data, there will definitely be some duplication.
>>>>
>>>> Thanks, vino.
>>>>
>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 下午12:01写道：
>>>>
>>>>> Hi Vino
>>>>> thank you for the information, actually I am using a trigger alert and
>>>>> processWindowFunction to send my results, but when my window slides or ends
>>>>> it sends again the objects and I an getting duplicated data
>>>>>
>>>>> El jue., 16 ago. 2018 a las 22:05, vino yang (<ya...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi Antonio,
>>>>>>
>>>>>> What results do not you want to get when creating each window?
>>>>>> Examples of the use of ProcessWindowFunction are included in many
>>>>>> test files in Flink's project, such as SideOutputITCase.scala or
>>>>>> WindowTranslationTest.scala.
>>>>>>
>>>>>> For more information on ProcessWindowFunction, you can refer to the
>>>>>> official website.[1]
>>>>>>
>>>>>> [1]:
>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>>>>>>
>>>>>> Thanks, vino.
>>>>>>
>>>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>> I am implementing a data stream where I use sliding windows but I am
>>>>>>> stuck because I need to set values to my object based on some if statements
>>>>>>> in my process function and send the object to the next step but I don't
>>>>>>> want results every time a window is creating
>>>>>>>
>>>>>>> if anyone has a good example on this that can help me
>>>>>>>
>>>>>>
Re: processWindowFunction
Posted by antonio saldivar <an...@gmail.com>.
Thank you fro the references
I have now my processFunction and getting the state but now how can i do
for the threshold times to group the elements and also as this is a global
window, how to purge because if going to keep increasing
El dom., 19 ago. 2018 a las 8:57, vino yang (<ya...@gmail.com>)
escribió:
> Hi antonio,
>
> Regarding your scenario, I think maybe you can consider using the
> ProcessFunction (or keyed ProcessFunction) function directly on the Stream.
> [1]
> It can handle each of your elements with a Timer, and you can combine
> Flink's state API[2] to store your data.
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations
> [2]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state
>
> Thanks, vino.
>
> antonio saldivar <an...@gmail.com> 于2018年8月19日周日 上午10:18写道：
>
>> hi Vino
>>
>> it is possible to use global window, then set the trigger onElement
>> comparing the element that has arrived with for example 10 mins, 20 mins
>> and 60 mins of data?
>>
>> I have rules evaluating sum of amount for 10,20 or 60 mins for the same
>> keyed element if the same id sum like $200 total within those thresholds
>> and count more or equals to 3 I need to be able to set some values to the
>> object if the object does not reach those thresholds i do not set the
>> values and keep sending the output with or without those value.
>>
>> just processing the object on the fly and send output
>>
>>
>>
>>
>>
>>
>>
>> El vie., 17 ago. 2018 a las 22:14, vino yang (<ya...@gmail.com>)
>> escribió:
>>
>>> Hi antonio,
>>>
>>> Yes, ProcessWindowFunction is a very low level window function.
>>> It allows you to access the data in the window and allows you to
>>> customize the output of the window.
>>> So if you use it, while giving you flexibility, you need to think about
>>> other things, which may require you to write more processing logic.
>>>
>>> Generally speaking, sliding windows usually have some data that is
>>> repeated, but a common mode is to apply a reduce function on it to get your
>>> calculation results.
>>> If you only send data, there will definitely be some duplication.
>>>
>>> Thanks, vino.
>>>
>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 下午12:01写道：
>>>
>>>> Hi Vino
>>>> thank you for the information, actually I am using a trigger alert and
>>>> processWindowFunction to send my results, but when my window slides or ends
>>>> it sends again the objects and I an getting duplicated data
>>>>
>>>> El jue., 16 ago. 2018 a las 22:05, vino yang (<ya...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi Antonio,
>>>>>
>>>>> What results do not you want to get when creating each window?
>>>>> Examples of the use of ProcessWindowFunction are included in many test
>>>>> files in Flink's project, such as SideOutputITCase.scala or
>>>>> WindowTranslationTest.scala.
>>>>>
>>>>> For more information on ProcessWindowFunction, you can refer to the
>>>>> official website.[1]
>>>>>
>>>>> [1]:
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>>>>>
>>>>> Thanks, vino.
>>>>>
>>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
>>>>>
>>>>>> Hello
>>>>>>
>>>>>> I am implementing a data stream where I use sliding windows but I am
>>>>>> stuck because I need to set values to my object based on some if statements
>>>>>> in my process function and send the object to the next step but I don't
>>>>>> want results every time a window is creating
>>>>>>
>>>>>> if anyone has a good example on this that can help me
>>>>>>
>>>>>
Re: processWindowFunction
Posted by vino yang <ya...@gmail.com>.
Hi antonio,
Regarding your scenario, I think maybe you can consider using the
ProcessFunction (or keyed ProcessFunction) function directly on the Stream.
[1]
It can handle each of your elements with a Timer, and you can combine
Flink's state API[2] to store your data.
[1]:
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations
[2]:
https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state
Thanks, vino.
antonio saldivar <an...@gmail.com> 于2018年8月19日周日 上午10:18写道：
> hi Vino
>
> it is possible to use global window, then set the trigger onElement
> comparing the element that has arrived with for example 10 mins, 20 mins
> and 60 mins of data?
>
> I have rules evaluating sum of amount for 10,20 or 60 mins for the same
> keyed element if the same id sum like $200 total within those thresholds
> and count more or equals to 3 I need to be able to set some values to the
> object if the object does not reach those thresholds i do not set the
> values and keep sending the output with or without those value.
>
> just processing the object on the fly and send output
>
>
>
>
>
>
>
> El vie., 17 ago. 2018 a las 22:14, vino yang (<ya...@gmail.com>)
> escribió:
>
>> Hi antonio,
>>
>> Yes, ProcessWindowFunction is a very low level window function.
>> It allows you to access the data in the window and allows you to
>> customize the output of the window.
>> So if you use it, while giving you flexibility, you need to think about
>> other things, which may require you to write more processing logic.
>>
>> Generally speaking, sliding windows usually have some data that is
>> repeated, but a common mode is to apply a reduce function on it to get your
>> calculation results.
>> If you only send data, there will definitely be some duplication.
>>
>> Thanks, vino.
>>
>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 下午12:01写道：
>>
>>> Hi Vino
>>> thank you for the information, actually I am using a trigger alert and
>>> processWindowFunction to send my results, but when my window slides or ends
>>> it sends again the objects and I an getting duplicated data
>>>
>>> El jue., 16 ago. 2018 a las 22:05, vino yang (<ya...@gmail.com>)
>>> escribió:
>>>
>>>> Hi Antonio,
>>>>
>>>> What results do not you want to get when creating each window?
>>>> Examples of the use of ProcessWindowFunction are included in many test
>>>> files in Flink's project, such as SideOutputITCase.scala or
>>>> WindowTranslationTest.scala.
>>>>
>>>> For more information on ProcessWindowFunction, you can refer to the
>>>> official website.[1]
>>>>
>>>> [1]:
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>>>>
>>>> Thanks, vino.
>>>>
>>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
>>>>
>>>>> Hello
>>>>>
>>>>> I am implementing a data stream where I use sliding windows but I am
>>>>> stuck because I need to set values to my object based on some if statements
>>>>> in my process function and send the object to the next step but I don't
>>>>> want results every time a window is creating
>>>>>
>>>>> if anyone has a good example on this that can help me
>>>>>
>>>>
Re: processWindowFunction
Posted by antonio saldivar <an...@gmail.com>.
hi Vino
it is possible to use global window, then set the trigger onElement
comparing the element that has arrived with for example 10 mins, 20 mins
and 60 mins of data?
I have rules evaluating sum of amount for 10,20 or 60 mins for the same
keyed element if the same id sum like $200 total within those thresholds
and count more or equals to 3 I need to be able to set some values to the
object if the object does not reach those thresholds i do not set the
values and keep sending the output with or without those value.
just processing the object on the fly and send output
El vie., 17 ago. 2018 a las 22:14, vino yang (<ya...@gmail.com>)
escribió:
> Hi antonio,
>
> Yes, ProcessWindowFunction is a very low level window function.
> It allows you to access the data in the window and allows you to customize
> the output of the window.
> So if you use it, while giving you flexibility, you need to think about
> other things, which may require you to write more processing logic.
>
> Generally speaking, sliding windows usually have some data that is
> repeated, but a common mode is to apply a reduce function on it to get your
> calculation results.
> If you only send data, there will definitely be some duplication.
>
> Thanks, vino.
>
> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 下午12:01写道：
>
>> Hi Vino
>> thank you for the information, actually I am using a trigger alert and
>> processWindowFunction to send my results, but when my window slides or ends
>> it sends again the objects and I an getting duplicated data
>>
>> El jue., 16 ago. 2018 a las 22:05, vino yang (<ya...@gmail.com>)
>> escribió:
>>
>>> Hi Antonio,
>>>
>>> What results do not you want to get when creating each window?
>>> Examples of the use of ProcessWindowFunction are included in many test
>>> files in Flink's project, such as SideOutputITCase.scala or
>>> WindowTranslationTest.scala.
>>>
>>> For more information on ProcessWindowFunction, you can refer to the
>>> official website.[1]
>>>
>>> [1]:
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>>>
>>> Thanks, vino.
>>>
>>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
>>>
>>>> Hello
>>>>
>>>> I am implementing a data stream where I use sliding windows but I am
>>>> stuck because I need to set values to my object based on some if statements
>>>> in my process function and send the object to the next step but I don't
>>>> want results every time a window is creating
>>>>
>>>> if anyone has a good example on this that can help me
>>>>
>>>
Re: processWindowFunction
Posted by vino yang <ya...@gmail.com>.
Hi antonio,
Yes, ProcessWindowFunction is a very low level window function.
It allows you to access the data in the window and allows you to customize
the output of the window.
So if you use it, while giving you flexibility, you need to think about
other things, which may require you to write more processing logic.
Generally speaking, sliding windows usually have some data that is
repeated, but a common mode is to apply a reduce function on it to get your
calculation results.
If you only send data, there will definitely be some duplication.
Thanks, vino.
antonio saldivar <an...@gmail.com> 于2018年8月17日周五 下午12:01写道：
> Hi Vino
> thank you for the information, actually I am using a trigger alert and
> processWindowFunction to send my results, but when my window slides or ends
> it sends again the objects and I an getting duplicated data
>
> El jue., 16 ago. 2018 a las 22:05, vino yang (<ya...@gmail.com>)
> escribió:
>
>> Hi Antonio,
>>
>> What results do not you want to get when creating each window?
>> Examples of the use of ProcessWindowFunction are included in many test
>> files in Flink's project, such as SideOutputITCase.scala or
>> WindowTranslationTest.scala.
>>
>> For more information on ProcessWindowFunction, you can refer to the
>> official website.[1]
>>
>> [1]:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>>
>> Thanks, vino.
>>
>> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
>>
>>> Hello
>>>
>>> I am implementing a data stream where I use sliding windows but I am
>>> stuck because I need to set values to my object based on some if statements
>>> in my process function and send the object to the next step but I don't
>>> want results every time a window is creating
>>>
>>> if anyone has a good example on this that can help me
>>>
>>
Re: processWindowFunction
Posted by antonio saldivar <an...@gmail.com>.
Hi Vino
thank you for the information, actually I am using a trigger alert and
processWindowFunction to send my results, but when my window slides or ends
it sends again the objects and I an getting duplicated data
El jue., 16 ago. 2018 a las 22:05, vino yang (<ya...@gmail.com>)
escribió:
> Hi Antonio,
>
> What results do not you want to get when creating each window?
> Examples of the use of ProcessWindowFunction are included in many test
> files in Flink's project, such as SideOutputITCase.scala or
> WindowTranslationTest.scala.
>
> For more information on ProcessWindowFunction, you can refer to the
> official website.[1]
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>
> Thanks, vino.
>
> antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
>
>> Hello
>>
>> I am implementing a data stream where I use sliding windows but I am
>> stuck because I need to set values to my object based on some if statements
>> in my process function and send the object to the next step but I don't
>> want results every time a window is creating
>>
>> if anyone has a good example on this that can help me
>>
>
Re: processWindowFunction
Posted by vino yang <ya...@gmail.com>.
Hi Antonio,
What results do not you want to get when creating each window?
Examples of the use of ProcessWindowFunction are included in many test
files in Flink's project, such as SideOutputITCase.scala or
WindowTranslationTest.scala.
For more information on ProcessWindowFunction, you can refer to the
official website.[1]
[1]:
https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
Thanks, vino.
antonio saldivar <an...@gmail.com> 于2018年8月17日周五 上午6:24写道：
> Hello
>
> I am implementing a data stream where I use sliding windows but I am stuck
> because I need to set values to my object based on some if statements in my
> process function and send the object to the next step but I don't want
> results every time a window is creating
>
> if anyone has a good example on this that can help me
>