You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by meneldor <me...@gmail.com> on 2021/01/14 14:44:46 UTC
Declaring and using ROW data type in PyFlink DataStream
Hello,
What is the correct way to use Python dict's as ROW type in pyflink? Im
trying this:
output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp' ],
[Types.STRING(), Types.STRING(),
Types.LONG() ])
class MyProcessFunction(KeyedProcessFunction):
def process_element(self, value, ctx:
'KeyedProcessFunction.Context', out: Collector):
result = {"id": ctx.get_current_key()[0], "data":
"some_string", "timestamp": 111111111111}
out.collect(result)
current_watermark = ctx.timer_service().current_watermark()
ctx.timer_service().register_event_time_timer(current_watermark + 1500)
def on_timer(self, timestamp, ctx:
'KeyedProcessFunction.OnTimerContext', out: 'Collector'):
logging.info(timestamp)
out.collect("On timer timestamp: " + str(timestamp))
ds.key_by(MyKeySelector(), key_type_info=Types.TUPLE([Types.STRING(),
Types.STRING()])) \
.process(MyProcessFunction(), output_type=output_type_info)
I just hardcoded the values in MyProcessFunction to be sure that the input
data doesnt mess the fields. So the data is correct but PyFlink trews an
exception:
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
> at
> org.apache.flink.api.java.typeutils.runtime.MaskUtils.readIntoMask(MaskUtils.java:73)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:202)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
> at
> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
> ... 10 more
However it works with primitive types like Types.STRING(). According
to the documentation the ROW type corresponds to the python's dict
type.
Regards
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by Xingbo Huang <hx...@gmail.com>.
Hi meneldor,
Yes. As the first version of Python DataStream, release-1.12 has not yet
covered all scenarios. In release-1.13, we will extend the function of
Python DataStream to cover most scenarios, and CoProcessFunction will
obviously be in it.
Best,
Xingbo
meneldor <me...@gmail.com> 于2021年1月19日周二 下午4:52写道:
> Thank you Xingbo!
>
> Do you plan to implement CoProcess functions too? Right now i cant find a
> convenient method to connect and merge two streams?
>
> Regards
>
> On Tue, Jan 19, 2021 at 4:16 AM Xingbo Huang <hx...@gmail.com> wrote:
>
>> Hi meneldor,
>>
>> 1. Yes. Although release 1.12.1 has not been officially released, it is
>> indeed available for download on PyPI.
>> In PyFlink 1.12.1, you only need to `yield` your output in `on_timer`.
>>
>> 2. Whenever an element comes, your `process_element` method will be
>> invoked, so you can directly get the `value` parameter in
>> `process_element`. The firing of the `on_timer` method depends on your
>> registering timer, as you wrote in the example
>> `ctx.timer_service().register_event_time_timer(current_watermark + 1500)`.
>> You might need state access[1] which will be supported in release-1.13. At
>> that time, you can get your state in `on_timer`, so as to conveniently
>> control the output.
>>
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-153%3A+Support+state+access+in+Python+DataStream+API
>> .
>>
>> Best,
>> Xingbo
>>
>> meneldor <me...@gmail.com> 于2021年1月18日周一 下午10:44写道:
>>
>>> Thank you Xingbo
>>>
>>> 1. I will try to use normal list instead of named. Thanks!
>>> 2. There is a new 1.12.1 version of pyflink which is using
>>> process_element(self, value, ctx: 'KeyedProcessFunction.Context')
>>>
>>> And what about the on_timer(self, timestamp, ctx:
>>> 'KeyedProcessFunction.OnTimerContext')? Can i access the value as in
>>> process_element() in the ctx for example?
>>>
>>> Thank you!
>>>
>>> On Mon, Jan 18, 2021 at 4:18 PM Xingbo Huang <hx...@gmail.com> wrote:
>>>
>>>> Hi Shuiqiang, meneldor,
>>>>
>>>> 1. In fact, there is a problem with using Python `Named Row` as the
>>>> return value of user-defined function in PyFlink.
>>>>
>>>> When serializing a Row data, the serializer of each field is consistent
>>>> with the order of the Row fields. But the field order of Python `Named Row`
>>>> has been sorted by field, and it was designed to better compare Named Row
>>>> and calculate hash values. So this can lead to
>>>> serialization/deserialization errors(The correspondence between serializer
>>>> and field is wrong). It is for performance considerations that serializers
>>>> are not specified according to file name, but `Named Row` support can be
>>>> achieved at the expense of a little performance for ease of use. For the
>>>> current example, I suggest returning a list or a normal Row, instead of a
>>>> Named Row.
>>>>
>>>>
>>>> 2. In pyflink 1.12.0, the method signature of `on_timer` should be `def
>>>> process_element(self, value, ctx: 'KeyedProcessFunction.Context', out:
>>>> Collector)`[1]. If you want to send data in `on_timer`, you can use
>>>> `Collector.collect`. e.g.
>>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context',
>>>> out: Collector):
>>>> out.collect(Row('a', 'b', 'c'))
>>>>
>>>> 3. >>> I am not sure if the timestamp field should be included in
>>>> output_type_info as i did now.
>>>>
>>>> If you return data with a time_stamp field, `output_type_info` needs to
>>>> have `time_stamp` field. For example, the data returned in your example
>>>> contains `time_stamp`, so your `output_type_info` needs to have the
>>>> information of this field.
>>>>
>>>>
>>>> [1]
>>>> https://github.com/apache/flink/blob/release-1.12.0/flink-python/pyflink/datastream/functions.py#L759
>>>>
>>>> Best,
>>>> Xingbo
>>>>
>>>> 2021年1月18日 下午9:21,meneldor <me...@gmail.com> 写道:
>>>>
>>>> Actually the *output_type_info* is ok, it was copy/paste typo. I
>>>> changed the function to:
>>>>
>>>> class MyProcessFunction(KeyedProcessFunction):
>>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
>>>> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=value[3][2], timestamp=ctx.timestamp())
>>>> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>>>>
>>>> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext'):
>>>> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=1111111111, timestamp=timestamp)
>>>>
>>>> And the type to:
>>>>
>>>> output_type_info = Types.ROW_NAMED(['id', 'tp', 'account', 'device_ts', 'timestamp'],
>>>> [Types.STRING(), Types.STRING(), Types.STRING(), Types.LONG(), Types.LONG()])
>>>>
>>>> I cant return the same data in *on_timer()* because there is no value
>>>> parameter. Thats why i hardcoded *device_ts*. However the exception
>>>> persists.
>>>>
>>>> I am not sure if the timestamp field should be included in *output_type_info* as i did now.
>>>>
>>>> Regards
>>>>
>>>>
>>>> On Mon, Jan 18, 2021 at 2:57 PM Shuiqiang Chen <ac...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi meneldor,
>>>>>
>>>>> Actually, the return type of the on_timer() must be the same as
>>>>> process_element(). It seems that the yield value of process_element() is
>>>>> missing the `timestamp` field. And the `output_type_info` has four field
>>>>> names but with 5 field types. Could you align them?
>>>>>
>>>>> Best,
>>>>> Shuiqiang
>>>>>
>>>>
>>>>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by meneldor <me...@gmail.com>.
Thank you Xingbo!
Do you plan to implement CoProcess functions too? Right now i cant find a
convenient method to connect and merge two streams?
Regards
On Tue, Jan 19, 2021 at 4:16 AM Xingbo Huang <hx...@gmail.com> wrote:
> Hi meneldor,
>
> 1. Yes. Although release 1.12.1 has not been officially released, it is
> indeed available for download on PyPI.
> In PyFlink 1.12.1, you only need to `yield` your output in `on_timer`.
>
> 2. Whenever an element comes, your `process_element` method will be
> invoked, so you can directly get the `value` parameter in
> `process_element`. The firing of the `on_timer` method depends on your
> registering timer, as you wrote in the example
> `ctx.timer_service().register_event_time_timer(current_watermark + 1500)`.
> You might need state access[1] which will be supported in release-1.13. At
> that time, you can get your state in `on_timer`, so as to conveniently
> control the output.
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-153%3A+Support+state+access+in+Python+DataStream+API
> .
>
> Best,
> Xingbo
>
> meneldor <me...@gmail.com> 于2021年1月18日周一 下午10:44写道:
>
>> Thank you Xingbo
>>
>> 1. I will try to use normal list instead of named. Thanks!
>> 2. There is a new 1.12.1 version of pyflink which is using
>> process_element(self, value, ctx: 'KeyedProcessFunction.Context')
>>
>> And what about the on_timer(self, timestamp, ctx:
>> 'KeyedProcessFunction.OnTimerContext')? Can i access the value as in
>> process_element() in the ctx for example?
>>
>> Thank you!
>>
>> On Mon, Jan 18, 2021 at 4:18 PM Xingbo Huang <hx...@gmail.com> wrote:
>>
>>> Hi Shuiqiang, meneldor,
>>>
>>> 1. In fact, there is a problem with using Python `Named Row` as the
>>> return value of user-defined function in PyFlink.
>>>
>>> When serializing a Row data, the serializer of each field is consistent
>>> with the order of the Row fields. But the field order of Python `Named Row`
>>> has been sorted by field, and it was designed to better compare Named Row
>>> and calculate hash values. So this can lead to
>>> serialization/deserialization errors(The correspondence between serializer
>>> and field is wrong). It is for performance considerations that serializers
>>> are not specified according to file name, but `Named Row` support can be
>>> achieved at the expense of a little performance for ease of use. For the
>>> current example, I suggest returning a list or a normal Row, instead of a
>>> Named Row.
>>>
>>>
>>> 2. In pyflink 1.12.0, the method signature of `on_timer` should be `def
>>> process_element(self, value, ctx: 'KeyedProcessFunction.Context', out:
>>> Collector)`[1]. If you want to send data in `on_timer`, you can use
>>> `Collector.collect`. e.g.
>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context',
>>> out: Collector):
>>> out.collect(Row('a', 'b', 'c'))
>>>
>>> 3. >>> I am not sure if the timestamp field should be included in
>>> output_type_info as i did now.
>>>
>>> If you return data with a time_stamp field, `output_type_info` needs to
>>> have `time_stamp` field. For example, the data returned in your example
>>> contains `time_stamp`, so your `output_type_info` needs to have the
>>> information of this field.
>>>
>>>
>>> [1]
>>> https://github.com/apache/flink/blob/release-1.12.0/flink-python/pyflink/datastream/functions.py#L759
>>>
>>> Best,
>>> Xingbo
>>>
>>> 2021年1月18日 下午9:21,meneldor <me...@gmail.com> 写道:
>>>
>>> Actually the *output_type_info* is ok, it was copy/paste typo. I
>>> changed the function to:
>>>
>>> class MyProcessFunction(KeyedProcessFunction):
>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
>>> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=value[3][2], timestamp=ctx.timestamp())
>>> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>>>
>>> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext'):
>>> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=1111111111, timestamp=timestamp)
>>>
>>> And the type to:
>>>
>>> output_type_info = Types.ROW_NAMED(['id', 'tp', 'account', 'device_ts', 'timestamp'],
>>> [Types.STRING(), Types.STRING(), Types.STRING(), Types.LONG(), Types.LONG()])
>>>
>>> I cant return the same data in *on_timer()* because there is no value
>>> parameter. Thats why i hardcoded *device_ts*. However the exception
>>> persists.
>>>
>>> I am not sure if the timestamp field should be included in *output_type_info* as i did now.
>>>
>>> Regards
>>>
>>>
>>> On Mon, Jan 18, 2021 at 2:57 PM Shuiqiang Chen <ac...@gmail.com>
>>> wrote:
>>>
>>>> Hi meneldor,
>>>>
>>>> Actually, the return type of the on_timer() must be the same as
>>>> process_element(). It seems that the yield value of process_element() is
>>>> missing the `timestamp` field. And the `output_type_info` has four field
>>>> names but with 5 field types. Could you align them?
>>>>
>>>> Best,
>>>> Shuiqiang
>>>>
>>>
>>>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by Xingbo Huang <hx...@gmail.com>.
Hi meneldor,
1. Yes. Although release 1.12.1 has not been officially released, it is
indeed available for download on PyPI.
In PyFlink 1.12.1, you only need to `yield` your output in `on_timer`.
2. Whenever an element comes, your `process_element` method will be
invoked, so you can directly get the `value` parameter in
`process_element`. The firing of the `on_timer` method depends on your
registering timer, as you wrote in the example
`ctx.timer_service().register_event_time_timer(current_watermark + 1500)`.
You might need state access[1] which will be supported in release-1.13. At
that time, you can get your state in `on_timer`, so as to conveniently
control the output.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-153%3A+Support+state+access+in+Python+DataStream+API
.
Best,
Xingbo
meneldor <me...@gmail.com> 于2021年1月18日周一 下午10:44写道:
> Thank you Xingbo
>
> 1. I will try to use normal list instead of named. Thanks!
> 2. There is a new 1.12.1 version of pyflink which is using process_element
> (self, value, ctx: 'KeyedProcessFunction.Context')
>
> And what about the on_timer(self, timestamp, ctx:
> 'KeyedProcessFunction.OnTimerContext')? Can i access the value as in
> process_element() in the ctx for example?
>
> Thank you!
>
> On Mon, Jan 18, 2021 at 4:18 PM Xingbo Huang <hx...@gmail.com> wrote:
>
>> Hi Shuiqiang, meneldor,
>>
>> 1. In fact, there is a problem with using Python `Named Row` as the
>> return value of user-defined function in PyFlink.
>>
>> When serializing a Row data, the serializer of each field is consistent
>> with the order of the Row fields. But the field order of Python `Named Row`
>> has been sorted by field, and it was designed to better compare Named Row
>> and calculate hash values. So this can lead to
>> serialization/deserialization errors(The correspondence between serializer
>> and field is wrong). It is for performance considerations that serializers
>> are not specified according to file name, but `Named Row` support can be
>> achieved at the expense of a little performance for ease of use. For the
>> current example, I suggest returning a list or a normal Row, instead of a
>> Named Row.
>>
>>
>> 2. In pyflink 1.12.0, the method signature of `on_timer` should be `def
>> process_element(self, value, ctx: 'KeyedProcessFunction.Context', out:
>> Collector)`[1]. If you want to send data in `on_timer`, you can use
>> `Collector.collect`. e.g.
>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context',
>> out: Collector):
>> out.collect(Row('a', 'b', 'c'))
>>
>> 3. >>> I am not sure if the timestamp field should be included in
>> output_type_info as i did now.
>>
>> If you return data with a time_stamp field, `output_type_info` needs to
>> have `time_stamp` field. For example, the data returned in your example
>> contains `time_stamp`, so your `output_type_info` needs to have the
>> information of this field.
>>
>>
>> [1]
>> https://github.com/apache/flink/blob/release-1.12.0/flink-python/pyflink/datastream/functions.py#L759
>>
>> Best,
>> Xingbo
>>
>> 2021年1月18日 下午9:21,meneldor <me...@gmail.com> 写道:
>>
>> Actually the *output_type_info* is ok, it was copy/paste typo. I changed
>> the function to:
>>
>> class MyProcessFunction(KeyedProcessFunction):
>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
>> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=value[3][2], timestamp=ctx.timestamp())
>> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>>
>> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext'):
>> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=1111111111, timestamp=timestamp)
>>
>> And the type to:
>>
>> output_type_info = Types.ROW_NAMED(['id', 'tp', 'account', 'device_ts', 'timestamp'],
>> [Types.STRING(), Types.STRING(), Types.STRING(), Types.LONG(), Types.LONG()])
>>
>> I cant return the same data in *on_timer()* because there is no value
>> parameter. Thats why i hardcoded *device_ts*. However the exception
>> persists.
>>
>> I am not sure if the timestamp field should be included in *output_type_info* as i did now.
>>
>> Regards
>>
>>
>> On Mon, Jan 18, 2021 at 2:57 PM Shuiqiang Chen <ac...@gmail.com>
>> wrote:
>>
>>> Hi meneldor,
>>>
>>> Actually, the return type of the on_timer() must be the same as
>>> process_element(). It seems that the yield value of process_element() is
>>> missing the `timestamp` field. And the `output_type_info` has four field
>>> names but with 5 field types. Could you align them?
>>>
>>> Best,
>>> Shuiqiang
>>>
>>
>>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by meneldor <me...@gmail.com>.
Thank you Xingbo
1. I will try to use normal list instead of named. Thanks!
2. There is a new 1.12.1 version of pyflink which is using process_element(
self, value, ctx: 'KeyedProcessFunction.Context')
And what about the on_timer(self, timestamp, ctx:
'KeyedProcessFunction.OnTimerContext')? Can i access the value as in
process_element() in the ctx for example?
Thank you!
On Mon, Jan 18, 2021 at 4:18 PM Xingbo Huang <hx...@gmail.com> wrote:
> Hi Shuiqiang, meneldor,
>
> 1. In fact, there is a problem with using Python `Named Row` as the return
> value of user-defined function in PyFlink.
>
> When serializing a Row data, the serializer of each field is consistent
> with the order of the Row fields. But the field order of Python `Named Row`
> has been sorted by field, and it was designed to better compare Named Row
> and calculate hash values. So this can lead to
> serialization/deserialization errors(The correspondence between serializer
> and field is wrong). It is for performance considerations that serializers
> are not specified according to file name, but `Named Row` support can be
> achieved at the expense of a little performance for ease of use. For the
> current example, I suggest returning a list or a normal Row, instead of a
> Named Row.
>
>
> 2. In pyflink 1.12.0, the method signature of `on_timer` should be `def
> process_element(self, value, ctx: 'KeyedProcessFunction.Context', out:
> Collector)`[1]. If you want to send data in `on_timer`, you can use
> `Collector.collect`. e.g.
> def process_element(self, value, ctx: 'KeyedProcessFunction.Context', out:
> Collector):
> out.collect(Row('a', 'b', 'c'))
>
> 3. >>> I am not sure if the timestamp field should be included in
> output_type_info as i did now.
>
> If you return data with a time_stamp field, `output_type_info` needs to
> have `time_stamp` field. For example, the data returned in your example
> contains `time_stamp`, so your `output_type_info` needs to have the
> information of this field.
>
>
> [1]
> https://github.com/apache/flink/blob/release-1.12.0/flink-python/pyflink/datastream/functions.py#L759
>
> Best,
> Xingbo
>
> 2021年1月18日 下午9:21,meneldor <me...@gmail.com> 写道:
>
> Actually the *output_type_info* is ok, it was copy/paste typo. I changed
> the function to:
>
> class MyProcessFunction(KeyedProcessFunction):
> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=value[3][2], timestamp=ctx.timestamp())
> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>
> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext'):
> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=1111111111, timestamp=timestamp)
>
> And the type to:
>
> output_type_info = Types.ROW_NAMED(['id', 'tp', 'account', 'device_ts', 'timestamp'],
> [Types.STRING(), Types.STRING(), Types.STRING(), Types.LONG(), Types.LONG()])
>
> I cant return the same data in *on_timer()* because there is no value
> parameter. Thats why i hardcoded *device_ts*. However the exception
> persists.
>
> I am not sure if the timestamp field should be included in *output_type_info* as i did now.
>
> Regards
>
>
> On Mon, Jan 18, 2021 at 2:57 PM Shuiqiang Chen <ac...@gmail.com>
> wrote:
>
>> Hi meneldor,
>>
>> Actually, the return type of the on_timer() must be the same as
>> process_element(). It seems that the yield value of process_element() is
>> missing the `timestamp` field. And the `output_type_info` has four field
>> names but with 5 field types. Could you align them?
>>
>> Best,
>> Shuiqiang
>>
>
>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by Xingbo Huang <hx...@gmail.com>.
Hi Shuiqiang, meneldor,
1. In fact, there is a problem with using Python `Named Row` as the return value of user-defined function in PyFlink.
When serializing a Row data, the serializer of each field is consistent with the order of the Row fields. But the field order of Python `Named Row` has been sorted by field, and it was designed to better compare Named Row and calculate hash values. So this can lead to serialization/deserialization errors(The correspondence between serializer and field is wrong). It is for performance considerations that serializers are not specified according to file name, but `Named Row` support can be achieved at the expense of a little performance for ease of use. For the current example, I suggest returning a list or a normal Row, instead of a Named Row.
2. In pyflink 1.12.0, the method signature of `on_timer` should be `def process_element(self, value, ctx: 'KeyedProcessFunction.Context', out: Collector)`[1]. If you want to send data in `on_timer`, you can use `Collector.collect`. e.g.
def process_element(self, value, ctx: 'KeyedProcessFunction.Context', out: Collector):
out.collect(Row('a', 'b', 'c'))
3. >>> I am not sure if the timestamp field should be included in output_type_info as i did now.
If you return data with a time_stamp field, `output_type_info` needs to have `time_stamp` field. For example, the data returned in your example contains `time_stamp`, so your `output_type_info` needs to have the information of this field.
[1] https://github.com/apache/flink/blob/release-1.12.0/flink-python/pyflink/datastream/functions.py#L759
Best,
Xingbo
> 2021年1月18日 下午9:21,meneldor <me...@gmail.com> 写道:
>
> Actually the output_type_info is ok, it was copy/paste typo. I changed the function to:
> class MyProcessFunction(KeyedProcessFunction):
> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=value[3][2], timestamp=ctx.timestamp())
> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>
> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext'):
> yield types.Row(id=ctx.get_current_key()[0], tp=ctx.get_current_key()[1], account="TEST", device_ts=1111111111, timestamp=timestamp)
> And the type to:
> output_type_info = Types.ROW_NAMED(['id', 'tp', 'account', 'device_ts', 'timestamp'],
> [Types.STRING(), Types.STRING(), Types.STRING(), Types.LONG(), Types.LONG()])
> I cant return the same data in on_timer() because there is no value parameter. Thats why i hardcoded device_ts. However the exception persists.
> I am not sure if the timestamp field should be included in output_type_info as i did now.
> Regards
>
> On Mon, Jan 18, 2021 at 2:57 PM Shuiqiang Chen <acqua.csq@gmail.com <ma...@gmail.com>> wrote:
> Hi meneldor,
>
> Actually, the return type of the on_timer() must be the same as process_element(). It seems that the yield value of process_element() is missing the `timestamp` field. And the `output_type_info` has four field names but with 5 field types. Could you align them?
>
> Best,
> Shuiqiang
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by meneldor <me...@gmail.com>.
Actually the *output_type_info* is ok, it was copy/paste typo. I changed
the function to:
class MyProcessFunction(KeyedProcessFunction):
def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
yield types.Row(id=ctx.get_current_key()[0],
tp=ctx.get_current_key()[1], account="TEST", device_ts=value[3][2],
timestamp=ctx.timestamp())
ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext'):
yield types.Row(id=ctx.get_current_key()[0],
tp=ctx.get_current_key()[1], account="TEST", device_ts=1111111111,
timestamp=timestamp)
And the type to:
output_type_info = Types.ROW_NAMED(['id', 'tp', 'account',
'device_ts', 'timestamp'],
[Types.STRING(), Types.STRING(),
Types.STRING(), Types.LONG(), Types.LONG()])
I cant return the same data in *on_timer()* because there is no value
parameter. Thats why i hardcoded *device_ts*. However the exception
persists.
I am not sure if the timestamp field should be included in
*output_type_info* as i did now.
Regards
On Mon, Jan 18, 2021 at 2:57 PM Shuiqiang Chen <ac...@gmail.com> wrote:
> Hi meneldor,
>
> Actually, the return type of the on_timer() must be the same as
> process_element(). It seems that the yield value of process_element() is
> missing the `timestamp` field. And the `output_type_info` has four field
> names but with 5 field types. Could you align them?
>
> Best,
> Shuiqiang
>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by Shuiqiang Chen <ac...@gmail.com>.
Hi meneldor,
Actually, the return type of the on_timer() must be the same as
process_element(). It seems that the yield value of process_element() is
missing the `timestamp` field. And the `output_type_info` has four field
names but with 5 field types. Could you align them?
Best,
Shuiqiang
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by meneldor <me...@gmail.com>.
Hi, here is the code. This is a JSON data from Maxwell CDC:
env = StreamExecutionEnvironment.get_execution_environment()
env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
env.get_config().set_auto_watermark_interval(2000)
env.set_parallelism(1)
device_type_info = Types.ROW_NAMED(['commit',
'ts',
'type',
'data',
'old'],
[Types.BOOLEAN(),
Types.LONG(),
Types.STRING(),
Types.ROW_NAMED(['id', 'tp',
'device_ts', 'account'],
[Types.STRING(),Types.STRING(),Types.LONG(),Types.STRING()]),
Types.ROW_NAMED(['id', 'tp',
'device_ts', 'account'],
[Types.STRING(),Types.STRING(),Types.LONG(),Types.STRING()])])
output_type_info = Types.ROW_NAMED(['id',
'tp',
'account',
'device_ts'
],
[Types.STRING(),
Types.STRING(),
Types.INT(),
Types.STRING(),
Types.LONG()
])
device_row_schema =
JsonRowDeserializationSchema.builder().type_info(device_type_info).build()
class KafkaRowTimestampAssigner(TimestampAssigner):
def extract_timestamp(self, value: Any, record_timestamp: int):
return int(value[3][2])
class MyKeySelector(KeySelector):
def get_key(self, value):
return (str(value[3][0]), str(value[3][1]))
class MyProcessFunction(KeyedProcessFunction):
def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
yield types.Row(id=ctx.get_current_key()[0],
tp=ctx.get_current_key()[1], account="TEST", device_ts=value[3][2])
ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext'):
yield types.Row(id=ctx.get_current_key()[0],
tp=ctx.get_current_key()[1], account="TEST", device_ts=1111111111,
timestamp=timestamp)
device_consumer = FlinkKafkaConsumer("device", device_row_schema,
{'bootstrap.servers': 'localhost:9092'})
device_consumer.set_start_from_earliest()
watermark_strategy =
WatermarkStrategy.for_monotonous_timestamps().with_timestamp_assigner(KafkaRowTimestampAssigner())
device_ds = env.add_source(device_consumer)
device_ds.assign_timestamps_and_watermarks(watermark_strategy).key_by(MyKeySelector(),
key_type_info=Types.TUPLE([Types.STRING(), Types.STRING()])) \
.process(MyProcessFunction(), output_type=output_type_info)
job_client = env.execute_async('Device enrichment Job')
job_client.get_job_execution_result().result()
This is the input data: {"commit": true, "ts": 1610546861, "type":
"update", "data": {"id": "id2", "tp": "B", "device_ts": 1610546861,
"account": "279"}, "old": {}}
1) If I change the output type to STRING() and return a str from
*process_element* everything is OK but I need to use
*JsonRowSerializationSchema* later on that data.
2) I'm not sure what to return in *on_timer* as it's missing the value
argument which *process_element* has.
Regards
On Mon, Jan 18, 2021 at 4:47 AM Shuiqiang Chen <ac...@gmail.com> wrote:
> Hi meneldor, Xingbo,
>
> Sorry for the late reply.
>
> Thanks a lot for Xingbo’s clarification.
>
> And according to the stacktrace of the exception, could you have a check
> whether the result data match the specified return type? BTW, please share
> your code if it’s ok, it will be of help to debug.
>
> Best,
> Shuiqiang
>
>
>
>
> meneldor <me...@gmail.com> 于2021年1月15日周五 下午4:59写道:
>
>> I imported pyflink.common.types.Row and used it as Shuiqiang suggested but
>> now Java throws a memory exception:
>>
>> Caused by: TimerException{java.lang.OutOfMemoryError: Java heap space}
>>> ... 11 more
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>> at
>>> org.apache.flink.table.runtime.util.SegmentsUtil.allocateReuseChars(SegmentsUtil.java:91)
>>> at
>>> org.apache.flink.table.runtime.util.StringUtf8Utils.decodeUTF8(StringUtf8Utils.java:127)
>>> at
>>> org.apache.flink.table.runtime.typeutils.serializers.python.StringSerializer.deserialize(StringSerializer.java:90)
>>> at
>>> org.apache.flink.table.runtime.typeutils.serializers.python.StringSerializer.deserialize(StringSerializer.java:41)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>> at
>>> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
>>> at
>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
>>> at
>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
>>> at
>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
>>> at
>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
>>> at
>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator$$Lambda$670/579781231.onProcessingTime(Unknown
>>> Source)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$17(StreamTask.java:1202)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$844/2129217743.run(Unknown
>>> Source)
>>>
>>
>> Regards
>>
>> On Fri, Jan 15, 2021 at 4:00 AM Xingbo Huang <hx...@gmail.com> wrote:
>>
>>> Hi meneldor,
>>>
>>> I guess Shuiqiang is not using the pyflink 1.12.0 to develop the
>>> example. The signature of the `process_element` method has been changed in
>>> the new version[1]. In pyflink 1.12.0, you can use `collector`.collect to
>>> send out your results.
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-20647
>>>
>>> Best,
>>> Xingbo
>>>
>>> meneldor <me...@gmail.com> 于2021年1月15日周五 上午1:20写道:
>>>
>>>> Thank you for the answer Shuiqiang!
>>>> Im using the last apache-flink version:
>>>>
>>>>> Requirement already up-to-date: apache-flink in
>>>>> ./venv/lib/python3.7/site-packages (1.12.0)
>>>>
>>>> however the method signature is using a collector:
>>>>
>>>> [image: image.png]
>>>> Im using the *setup-pyflink-virtual-env.sh* shell script from the
>>>> docs(which uses pip).
>>>>
>>>> Regards
>>>>
>>>> On Thu, Jan 14, 2021 at 6:47 PM Shuiqiang Chen <ac...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi meneldor,
>>>>>
>>>>> The main cause of the error is that there is a bug in
>>>>> `ctx.timer_service().current_watermark()`. At the beginning the stream,
>>>>> when the first record come into the KeyedProcessFunction.process_element()
>>>>> , the current_watermark will be the Long.MIN_VALUE at Java side, while at
>>>>> the Python side, it becomes LONG.MAX_VALUE which is 9223372036854775807.
>>>>>
>>>>> >>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>>>>>
>>>>> Here, 9223372036854775807 + 1500 is 9223372036854777307 which will be
>>>>> automatically converted to a long interger in python but will cause Long
>>>>> value overflow in Java when deserializing the registered timer value. I
>>>>> will craete a issue to fix the bug.
>>>>>
>>>>> Let’s return to your initial question, at PyFlink you could create a
>>>>> Row Type data as bellow:
>>>>>
>>>>> >>> row_data = Row(id=‘my id’, data=’some data’, timestamp=1111)
>>>>>
>>>>> And I wonder which release version of flink the code snippet you
>>>>> provided based on? The latest API for
>>>>> KeyedProcessFunction.process_element() and KeyedProcessFunction.on_timer()
>>>>> will not provid a `collector` to collect output data but use `yield` which
>>>>> is a more pythonic approach.
>>>>>
>>>>> Please refer to the following code:
>>>>>
>>>>> def keyed_process_function_example():
>>>>> env = StreamExecutionEnvironment.get_execution_environment()
>>>>> env.set_parallelism(1)
>>>>> env.get_config().set_auto_watermark_interval(2000)
>>>>> env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
>>>>> data_stream = env.from_collection([(1, 'hello', '1603708211000'),
>>>>> (2, 'hi', '1603708224000'),
>>>>> (3, 'hello', '1603708226000'),
>>>>> (4, 'hi', '1603708289000')],
>>>>> type_info=Types.ROW([Types.INT(), Types.STRING(), Types.STRING()]))
>>>>>
>>>>> class MyTimestampAssigner(TimestampAssigner):
>>>>>
>>>>> def extract_timestamp(self, value, record_timestamp) -> int:
>>>>> return int(value[2])
>>>>>
>>>>> class MyProcessFunction(KeyedProcessFunction):
>>>>>
>>>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
>>>>> yield Row(id=ctx.get_current_key()[1], data='some_string', timestamp=11111111)
>>>>> # current_watermark = ctx.timer_service().current_watermark()
>>>>> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>>>>>
>>>>> def on_timer(self, timestamp: int, ctx: 'KeyedProcessFunction.OnTimerContext'):
>>>>> yield Row(id=ctx.get_current_key()[1], data='current on timer timestamp: ' + str(timestamp),
>>>>> timestamp=timestamp)
>>>>>
>>>>> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp'], [Types.STRING(), Types.STRING(), Types.INT()])
>>>>> watermark_strategy = WatermarkStrategy.for_monotonous_timestamps() \
>>>>> .with_timestamp_assigner(MyTimestampAssigner())
>>>>> data_stream.assign_timestamps_and_watermarks(watermark_strategy) \
>>>>> .key_by(lambda x: (x[0], x[1]), key_type_info=Types.TUPLE([Types.INT(), Types.STRING()])) \
>>>>> .process(MyProcessFunction(), output_type=output_type_info).print()
>>>>> env.execute('test keyed process function')
>>>>>
>>>>>
>>>>> Best,
>>>>> Shuiqiang
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> meneldor <me...@gmail.com> 于2021年1月14日周四 下午10:45写道:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> What is the correct way to use Python dict's as ROW type in pyflink?
>>>>>> Im trying this:
>>>>>>
>>>>>> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp' ],
>>>>>> [Types.STRING(), Types.STRING(), Types.LONG() ])
>>>>>>
>>>>>> class MyProcessFunction(KeyedProcessFunction):
>>>>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context', out: Collector):
>>>>>> result = {"id": ctx.get_current_key()[0], "data": "some_string", "timestamp": 111111111111}
>>>>>> out.collect(result)
>>>>>> current_watermark = ctx.timer_service().current_watermark()
>>>>>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>>>>>>
>>>>>> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext', out: 'Collector'):
>>>>>> logging.info(timestamp)
>>>>>> out.collect("On timer timestamp: " + str(timestamp))
>>>>>>
>>>>>> ds.key_by(MyKeySelector(), key_type_info=Types.TUPLE([Types.STRING(), Types.STRING()])) \
>>>>>> .process(MyProcessFunction(), output_type=output_type_info)
>>>>>>
>>>>>>
>>>>>> I just hardcoded the values in MyProcessFunction to be sure that the
>>>>>> input data doesnt mess the fields. So the data is correct but PyFlink trews
>>>>>> an exception:
>>>>>>
>>>>>> at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
>>>>>>> at
>>>>>>> org.apache.flink.api.java.typeutils.runtime.MaskUtils.readIntoMask(MaskUtils.java:73)
>>>>>>> at
>>>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:202)
>>>>>>> at
>>>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>>>>>> at
>>>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>>>>>>> at
>>>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>>>>>> at
>>>>>>> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
>>>>>>> at
>>>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
>>>>>>> at
>>>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
>>>>>>> at
>>>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
>>>>>>> at
>>>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
>>>>>>> at
>>>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
>>>>>>> ... 10 more
>>>>>>
>>>>>> However it works with primitive types like Types.STRING(). According to the documentation the ROW type corresponds to the python's dict type.
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by Shuiqiang Chen <ac...@gmail.com>.
Hi meneldor, Xingbo,
Sorry for the late reply.
Thanks a lot for Xingbo’s clarification.
And according to the stacktrace of the exception, could you have a check
whether the result data match the specified return type? BTW, please share
your code if it’s ok, it will be of help to debug.
Best,
Shuiqiang
meneldor <me...@gmail.com> 于2021年1月15日周五 下午4:59写道:
> I imported pyflink.common.types.Row and used it as Shuiqiang suggested but
> now Java throws a memory exception:
>
> Caused by: TimerException{java.lang.OutOfMemoryError: Java heap space}
>> ... 11 more
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.flink.table.runtime.util.SegmentsUtil.allocateReuseChars(SegmentsUtil.java:91)
>> at
>> org.apache.flink.table.runtime.util.StringUtf8Utils.decodeUTF8(StringUtf8Utils.java:127)
>> at
>> org.apache.flink.table.runtime.typeutils.serializers.python.StringSerializer.deserialize(StringSerializer.java:90)
>> at
>> org.apache.flink.table.runtime.typeutils.serializers.python.StringSerializer.deserialize(StringSerializer.java:41)
>> at
>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>> at
>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>> at
>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>> at
>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>> at
>> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
>> at
>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
>> at
>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
>> at
>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
>> at
>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
>> at
>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator$$Lambda$670/579781231.onProcessingTime(Unknown
>> Source)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$17(StreamTask.java:1202)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$844/2129217743.run(Unknown
>> Source)
>>
>
> Regards
>
> On Fri, Jan 15, 2021 at 4:00 AM Xingbo Huang <hx...@gmail.com> wrote:
>
>> Hi meneldor,
>>
>> I guess Shuiqiang is not using the pyflink 1.12.0 to develop the example.
>> The signature of the `process_element` method has been changed in the new
>> version[1]. In pyflink 1.12.0, you can use `collector`.collect to send out
>> your results.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-20647
>>
>> Best,
>> Xingbo
>>
>> meneldor <me...@gmail.com> 于2021年1月15日周五 上午1:20写道:
>>
>>> Thank you for the answer Shuiqiang!
>>> Im using the last apache-flink version:
>>>
>>>> Requirement already up-to-date: apache-flink in
>>>> ./venv/lib/python3.7/site-packages (1.12.0)
>>>
>>> however the method signature is using a collector:
>>>
>>> [image: image.png]
>>> Im using the *setup-pyflink-virtual-env.sh* shell script from the
>>> docs(which uses pip).
>>>
>>> Regards
>>>
>>> On Thu, Jan 14, 2021 at 6:47 PM Shuiqiang Chen <ac...@gmail.com>
>>> wrote:
>>>
>>>> Hi meneldor,
>>>>
>>>> The main cause of the error is that there is a bug in
>>>> `ctx.timer_service().current_watermark()`. At the beginning the stream,
>>>> when the first record come into the KeyedProcessFunction.process_element()
>>>> , the current_watermark will be the Long.MIN_VALUE at Java side, while at
>>>> the Python side, it becomes LONG.MAX_VALUE which is 9223372036854775807.
>>>>
>>>> >>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>>>>
>>>> Here, 9223372036854775807 + 1500 is 9223372036854777307 which will be
>>>> automatically converted to a long interger in python but will cause Long
>>>> value overflow in Java when deserializing the registered timer value. I
>>>> will craete a issue to fix the bug.
>>>>
>>>> Let’s return to your initial question, at PyFlink you could create a
>>>> Row Type data as bellow:
>>>>
>>>> >>> row_data = Row(id=‘my id’, data=’some data’, timestamp=1111)
>>>>
>>>> And I wonder which release version of flink the code snippet you
>>>> provided based on? The latest API for
>>>> KeyedProcessFunction.process_element() and KeyedProcessFunction.on_timer()
>>>> will not provid a `collector` to collect output data but use `yield` which
>>>> is a more pythonic approach.
>>>>
>>>> Please refer to the following code:
>>>>
>>>> def keyed_process_function_example():
>>>> env = StreamExecutionEnvironment.get_execution_environment()
>>>> env.set_parallelism(1)
>>>> env.get_config().set_auto_watermark_interval(2000)
>>>> env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
>>>> data_stream = env.from_collection([(1, 'hello', '1603708211000'),
>>>> (2, 'hi', '1603708224000'),
>>>> (3, 'hello', '1603708226000'),
>>>> (4, 'hi', '1603708289000')],
>>>> type_info=Types.ROW([Types.INT(), Types.STRING(), Types.STRING()]))
>>>>
>>>> class MyTimestampAssigner(TimestampAssigner):
>>>>
>>>> def extract_timestamp(self, value, record_timestamp) -> int:
>>>> return int(value[2])
>>>>
>>>> class MyProcessFunction(KeyedProcessFunction):
>>>>
>>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
>>>> yield Row(id=ctx.get_current_key()[1], data='some_string', timestamp=11111111)
>>>> # current_watermark = ctx.timer_service().current_watermark()
>>>> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>>>>
>>>> def on_timer(self, timestamp: int, ctx: 'KeyedProcessFunction.OnTimerContext'):
>>>> yield Row(id=ctx.get_current_key()[1], data='current on timer timestamp: ' + str(timestamp),
>>>> timestamp=timestamp)
>>>>
>>>> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp'], [Types.STRING(), Types.STRING(), Types.INT()])
>>>> watermark_strategy = WatermarkStrategy.for_monotonous_timestamps() \
>>>> .with_timestamp_assigner(MyTimestampAssigner())
>>>> data_stream.assign_timestamps_and_watermarks(watermark_strategy) \
>>>> .key_by(lambda x: (x[0], x[1]), key_type_info=Types.TUPLE([Types.INT(), Types.STRING()])) \
>>>> .process(MyProcessFunction(), output_type=output_type_info).print()
>>>> env.execute('test keyed process function')
>>>>
>>>>
>>>> Best,
>>>> Shuiqiang
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> meneldor <me...@gmail.com> 于2021年1月14日周四 下午10:45写道:
>>>>
>>>>> Hello,
>>>>>
>>>>> What is the correct way to use Python dict's as ROW type in pyflink?
>>>>> Im trying this:
>>>>>
>>>>> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp' ],
>>>>> [Types.STRING(), Types.STRING(), Types.LONG() ])
>>>>>
>>>>> class MyProcessFunction(KeyedProcessFunction):
>>>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context', out: Collector):
>>>>> result = {"id": ctx.get_current_key()[0], "data": "some_string", "timestamp": 111111111111}
>>>>> out.collect(result)
>>>>> current_watermark = ctx.timer_service().current_watermark()
>>>>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>>>>>
>>>>> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext', out: 'Collector'):
>>>>> logging.info(timestamp)
>>>>> out.collect("On timer timestamp: " + str(timestamp))
>>>>>
>>>>> ds.key_by(MyKeySelector(), key_type_info=Types.TUPLE([Types.STRING(), Types.STRING()])) \
>>>>> .process(MyProcessFunction(), output_type=output_type_info)
>>>>>
>>>>>
>>>>> I just hardcoded the values in MyProcessFunction to be sure that the
>>>>> input data doesnt mess the fields. So the data is correct but PyFlink trews
>>>>> an exception:
>>>>>
>>>>> at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
>>>>>> at
>>>>>> org.apache.flink.api.java.typeutils.runtime.MaskUtils.readIntoMask(MaskUtils.java:73)
>>>>>> at
>>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:202)
>>>>>> at
>>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>>>>> at
>>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>>>>>> at
>>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>>>>> at
>>>>>> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
>>>>>> at
>>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
>>>>>> at
>>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
>>>>>> at
>>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
>>>>>> at
>>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
>>>>>> at
>>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
>>>>>> ... 10 more
>>>>>
>>>>> However it works with primitive types like Types.STRING(). According to the documentation the ROW type corresponds to the python's dict type.
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by meneldor <me...@gmail.com>.
I imported pyflink.common.types.Row and used it as Shuiqiang suggested but
now Java throws a memory exception:
Caused by: TimerException{java.lang.OutOfMemoryError: Java heap space}
> ... 11 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.flink.table.runtime.util.SegmentsUtil.allocateReuseChars(SegmentsUtil.java:91)
> at
> org.apache.flink.table.runtime.util.StringUtf8Utils.decodeUTF8(StringUtf8Utils.java:127)
> at
> org.apache.flink.table.runtime.typeutils.serializers.python.StringSerializer.deserialize(StringSerializer.java:90)
> at
> org.apache.flink.table.runtime.typeutils.serializers.python.StringSerializer.deserialize(StringSerializer.java:41)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
> at
> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator$$Lambda$670/579781231.onProcessingTime(Unknown
> Source)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$17(StreamTask.java:1202)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$844/2129217743.run(Unknown
> Source)
>
Regards
On Fri, Jan 15, 2021 at 4:00 AM Xingbo Huang <hx...@gmail.com> wrote:
> Hi meneldor,
>
> I guess Shuiqiang is not using the pyflink 1.12.0 to develop the example.
> The signature of the `process_element` method has been changed in the new
> version[1]. In pyflink 1.12.0, you can use `collector`.collect to send out
> your results.
>
> [1] https://issues.apache.org/jira/browse/FLINK-20647
>
> Best,
> Xingbo
>
> meneldor <me...@gmail.com> 于2021年1月15日周五 上午1:20写道:
>
>> Thank you for the answer Shuiqiang!
>> Im using the last apache-flink version:
>>
>>> Requirement already up-to-date: apache-flink in
>>> ./venv/lib/python3.7/site-packages (1.12.0)
>>
>> however the method signature is using a collector:
>>
>> [image: image.png]
>> Im using the *setup-pyflink-virtual-env.sh* shell script from the
>> docs(which uses pip).
>>
>> Regards
>>
>> On Thu, Jan 14, 2021 at 6:47 PM Shuiqiang Chen <ac...@gmail.com>
>> wrote:
>>
>>> Hi meneldor,
>>>
>>> The main cause of the error is that there is a bug in
>>> `ctx.timer_service().current_watermark()`. At the beginning the stream,
>>> when the first record come into the KeyedProcessFunction.process_element()
>>> , the current_watermark will be the Long.MIN_VALUE at Java side, while at
>>> the Python side, it becomes LONG.MAX_VALUE which is 9223372036854775807.
>>>
>>> >>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>>>
>>> Here, 9223372036854775807 + 1500 is 9223372036854777307 which will be
>>> automatically converted to a long interger in python but will cause Long
>>> value overflow in Java when deserializing the registered timer value. I
>>> will craete a issue to fix the bug.
>>>
>>> Let’s return to your initial question, at PyFlink you could create a Row
>>> Type data as bellow:
>>>
>>> >>> row_data = Row(id=‘my id’, data=’some data’, timestamp=1111)
>>>
>>> And I wonder which release version of flink the code snippet you
>>> provided based on? The latest API for
>>> KeyedProcessFunction.process_element() and KeyedProcessFunction.on_timer()
>>> will not provid a `collector` to collect output data but use `yield` which
>>> is a more pythonic approach.
>>>
>>> Please refer to the following code:
>>>
>>> def keyed_process_function_example():
>>> env = StreamExecutionEnvironment.get_execution_environment()
>>> env.set_parallelism(1)
>>> env.get_config().set_auto_watermark_interval(2000)
>>> env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
>>> data_stream = env.from_collection([(1, 'hello', '1603708211000'),
>>> (2, 'hi', '1603708224000'),
>>> (3, 'hello', '1603708226000'),
>>> (4, 'hi', '1603708289000')],
>>> type_info=Types.ROW([Types.INT(), Types.STRING(), Types.STRING()]))
>>>
>>> class MyTimestampAssigner(TimestampAssigner):
>>>
>>> def extract_timestamp(self, value, record_timestamp) -> int:
>>> return int(value[2])
>>>
>>> class MyProcessFunction(KeyedProcessFunction):
>>>
>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
>>> yield Row(id=ctx.get_current_key()[1], data='some_string', timestamp=11111111)
>>> # current_watermark = ctx.timer_service().current_watermark()
>>> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>>>
>>> def on_timer(self, timestamp: int, ctx: 'KeyedProcessFunction.OnTimerContext'):
>>> yield Row(id=ctx.get_current_key()[1], data='current on timer timestamp: ' + str(timestamp),
>>> timestamp=timestamp)
>>>
>>> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp'], [Types.STRING(), Types.STRING(), Types.INT()])
>>> watermark_strategy = WatermarkStrategy.for_monotonous_timestamps() \
>>> .with_timestamp_assigner(MyTimestampAssigner())
>>> data_stream.assign_timestamps_and_watermarks(watermark_strategy) \
>>> .key_by(lambda x: (x[0], x[1]), key_type_info=Types.TUPLE([Types.INT(), Types.STRING()])) \
>>> .process(MyProcessFunction(), output_type=output_type_info).print()
>>> env.execute('test keyed process function')
>>>
>>>
>>> Best,
>>> Shuiqiang
>>>
>>>
>>>
>>>
>>>
>>> meneldor <me...@gmail.com> 于2021年1月14日周四 下午10:45写道:
>>>
>>>> Hello,
>>>>
>>>> What is the correct way to use Python dict's as ROW type in pyflink? Im
>>>> trying this:
>>>>
>>>> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp' ],
>>>> [Types.STRING(), Types.STRING(), Types.LONG() ])
>>>>
>>>> class MyProcessFunction(KeyedProcessFunction):
>>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context', out: Collector):
>>>> result = {"id": ctx.get_current_key()[0], "data": "some_string", "timestamp": 111111111111}
>>>> out.collect(result)
>>>> current_watermark = ctx.timer_service().current_watermark()
>>>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>>>>
>>>> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext', out: 'Collector'):
>>>> logging.info(timestamp)
>>>> out.collect("On timer timestamp: " + str(timestamp))
>>>>
>>>> ds.key_by(MyKeySelector(), key_type_info=Types.TUPLE([Types.STRING(), Types.STRING()])) \
>>>> .process(MyProcessFunction(), output_type=output_type_info)
>>>>
>>>>
>>>> I just hardcoded the values in MyProcessFunction to be sure that the
>>>> input data doesnt mess the fields. So the data is correct but PyFlink trews
>>>> an exception:
>>>>
>>>> at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
>>>>> at
>>>>> org.apache.flink.api.java.typeutils.runtime.MaskUtils.readIntoMask(MaskUtils.java:73)
>>>>> at
>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:202)
>>>>> at
>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>>>> at
>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>>>>> at
>>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>>>> at
>>>>> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
>>>>> at
>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
>>>>> at
>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
>>>>> at
>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
>>>>> at
>>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
>>>>> at
>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
>>>>> ... 10 more
>>>>
>>>> However it works with primitive types like Types.STRING(). According to the documentation the ROW type corresponds to the python's dict type.
>>>>
>>>>
>>>> Regards
>>>>
>>>>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by Xingbo Huang <hx...@gmail.com>.
Hi meneldor,
I guess Shuiqiang is not using the pyflink 1.12.0 to develop the example.
The signature of the `process_element` method has been changed in the new
version[1]. In pyflink 1.12.0, you can use `collector`.collect to send out
your results.
[1] https://issues.apache.org/jira/browse/FLINK-20647
Best,
Xingbo
meneldor <me...@gmail.com> 于2021年1月15日周五 上午1:20写道:
> Thank you for the answer Shuiqiang!
> Im using the last apache-flink version:
>
>> Requirement already up-to-date: apache-flink in
>> ./venv/lib/python3.7/site-packages (1.12.0)
>
> however the method signature is using a collector:
>
> [image: image.png]
> Im using the *setup-pyflink-virtual-env.sh* shell script from the
> docs(which uses pip).
>
> Regards
>
> On Thu, Jan 14, 2021 at 6:47 PM Shuiqiang Chen <ac...@gmail.com>
> wrote:
>
>> Hi meneldor,
>>
>> The main cause of the error is that there is a bug in
>> `ctx.timer_service().current_watermark()`. At the beginning the stream,
>> when the first record come into the KeyedProcessFunction.process_element()
>> , the current_watermark will be the Long.MIN_VALUE at Java side, while at
>> the Python side, it becomes LONG.MAX_VALUE which is 9223372036854775807.
>>
>> >>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>>
>> Here, 9223372036854775807 + 1500 is 9223372036854777307 which will be
>> automatically converted to a long interger in python but will cause Long
>> value overflow in Java when deserializing the registered timer value. I
>> will craete a issue to fix the bug.
>>
>> Let’s return to your initial question, at PyFlink you could create a Row
>> Type data as bellow:
>>
>> >>> row_data = Row(id=‘my id’, data=’some data’, timestamp=1111)
>>
>> And I wonder which release version of flink the code snippet you provided
>> based on? The latest API for KeyedProcessFunction.process_element() and
>> KeyedProcessFunction.on_timer() will not provid a `collector` to collect
>> output data but use `yield` which is a more pythonic approach.
>>
>> Please refer to the following code:
>>
>> def keyed_process_function_example():
>> env = StreamExecutionEnvironment.get_execution_environment()
>> env.set_parallelism(1)
>> env.get_config().set_auto_watermark_interval(2000)
>> env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
>> data_stream = env.from_collection([(1, 'hello', '1603708211000'),
>> (2, 'hi', '1603708224000'),
>> (3, 'hello', '1603708226000'),
>> (4, 'hi', '1603708289000')],
>> type_info=Types.ROW([Types.INT(), Types.STRING(), Types.STRING()]))
>>
>> class MyTimestampAssigner(TimestampAssigner):
>>
>> def extract_timestamp(self, value, record_timestamp) -> int:
>> return int(value[2])
>>
>> class MyProcessFunction(KeyedProcessFunction):
>>
>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
>> yield Row(id=ctx.get_current_key()[1], data='some_string', timestamp=11111111)
>> # current_watermark = ctx.timer_service().current_watermark()
>> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>>
>> def on_timer(self, timestamp: int, ctx: 'KeyedProcessFunction.OnTimerContext'):
>> yield Row(id=ctx.get_current_key()[1], data='current on timer timestamp: ' + str(timestamp),
>> timestamp=timestamp)
>>
>> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp'], [Types.STRING(), Types.STRING(), Types.INT()])
>> watermark_strategy = WatermarkStrategy.for_monotonous_timestamps() \
>> .with_timestamp_assigner(MyTimestampAssigner())
>> data_stream.assign_timestamps_and_watermarks(watermark_strategy) \
>> .key_by(lambda x: (x[0], x[1]), key_type_info=Types.TUPLE([Types.INT(), Types.STRING()])) \
>> .process(MyProcessFunction(), output_type=output_type_info).print()
>> env.execute('test keyed process function')
>>
>>
>> Best,
>> Shuiqiang
>>
>>
>>
>>
>>
>> meneldor <me...@gmail.com> 于2021年1月14日周四 下午10:45写道:
>>
>>> Hello,
>>>
>>> What is the correct way to use Python dict's as ROW type in pyflink? Im
>>> trying this:
>>>
>>> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp' ],
>>> [Types.STRING(), Types.STRING(), Types.LONG() ])
>>>
>>> class MyProcessFunction(KeyedProcessFunction):
>>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context', out: Collector):
>>> result = {"id": ctx.get_current_key()[0], "data": "some_string", "timestamp": 111111111111}
>>> out.collect(result)
>>> current_watermark = ctx.timer_service().current_watermark()
>>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>>>
>>> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext', out: 'Collector'):
>>> logging.info(timestamp)
>>> out.collect("On timer timestamp: " + str(timestamp))
>>>
>>> ds.key_by(MyKeySelector(), key_type_info=Types.TUPLE([Types.STRING(), Types.STRING()])) \
>>> .process(MyProcessFunction(), output_type=output_type_info)
>>>
>>>
>>> I just hardcoded the values in MyProcessFunction to be sure that the
>>> input data doesnt mess the fields. So the data is correct but PyFlink trews
>>> an exception:
>>>
>>> at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
>>>> at
>>>> org.apache.flink.api.java.typeutils.runtime.MaskUtils.readIntoMask(MaskUtils.java:73)
>>>> at
>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:202)
>>>> at
>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>>> at
>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>>>> at
>>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>>> at
>>>> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
>>>> at
>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
>>>> at
>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
>>>> at
>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
>>>> at
>>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
>>>> at
>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
>>>> ... 10 more
>>>
>>> However it works with primitive types like Types.STRING(). According to the documentation the ROW type corresponds to the python's dict type.
>>>
>>>
>>> Regards
>>>
>>>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by meneldor <me...@gmail.com>.
Thank you for the answer Shuiqiang!
Im using the last apache-flink version:
> Requirement already up-to-date: apache-flink in
> ./venv/lib/python3.7/site-packages (1.12.0)
however the method signature is using a collector:
[image: image.png]
Im using the *setup-pyflink-virtual-env.sh* shell script from the
docs(which uses pip).
Regards
On Thu, Jan 14, 2021 at 6:47 PM Shuiqiang Chen <ac...@gmail.com> wrote:
> Hi meneldor,
>
> The main cause of the error is that there is a bug in
> `ctx.timer_service().current_watermark()`. At the beginning the stream,
> when the first record come into the KeyedProcessFunction.process_element()
> , the current_watermark will be the Long.MIN_VALUE at Java side, while at
> the Python side, it becomes LONG.MAX_VALUE which is 9223372036854775807.
>
> >>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>
> Here, 9223372036854775807 + 1500 is 9223372036854777307 which will be
> automatically converted to a long interger in python but will cause Long
> value overflow in Java when deserializing the registered timer value. I
> will craete a issue to fix the bug.
>
> Let’s return to your initial question, at PyFlink you could create a Row
> Type data as bellow:
>
> >>> row_data = Row(id=‘my id’, data=’some data’, timestamp=1111)
>
> And I wonder which release version of flink the code snippet you provided
> based on? The latest API for KeyedProcessFunction.process_element() and
> KeyedProcessFunction.on_timer() will not provid a `collector` to collect
> output data but use `yield` which is a more pythonic approach.
>
> Please refer to the following code:
>
> def keyed_process_function_example():
> env = StreamExecutionEnvironment.get_execution_environment()
> env.set_parallelism(1)
> env.get_config().set_auto_watermark_interval(2000)
> env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
> data_stream = env.from_collection([(1, 'hello', '1603708211000'),
> (2, 'hi', '1603708224000'),
> (3, 'hello', '1603708226000'),
> (4, 'hi', '1603708289000')],
> type_info=Types.ROW([Types.INT(), Types.STRING(), Types.STRING()]))
>
> class MyTimestampAssigner(TimestampAssigner):
>
> def extract_timestamp(self, value, record_timestamp) -> int:
> return int(value[2])
>
> class MyProcessFunction(KeyedProcessFunction):
>
> def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
> yield Row(id=ctx.get_current_key()[1], data='some_string', timestamp=11111111)
> # current_watermark = ctx.timer_service().current_watermark()
> ctx.timer_service().register_event_time_timer(ctx.timestamp() + 1500)
>
> def on_timer(self, timestamp: int, ctx: 'KeyedProcessFunction.OnTimerContext'):
> yield Row(id=ctx.get_current_key()[1], data='current on timer timestamp: ' + str(timestamp),
> timestamp=timestamp)
>
> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp'], [Types.STRING(), Types.STRING(), Types.INT()])
> watermark_strategy = WatermarkStrategy.for_monotonous_timestamps() \
> .with_timestamp_assigner(MyTimestampAssigner())
> data_stream.assign_timestamps_and_watermarks(watermark_strategy) \
> .key_by(lambda x: (x[0], x[1]), key_type_info=Types.TUPLE([Types.INT(), Types.STRING()])) \
> .process(MyProcessFunction(), output_type=output_type_info).print()
> env.execute('test keyed process function')
>
>
> Best,
> Shuiqiang
>
>
>
>
>
> meneldor <me...@gmail.com> 于2021年1月14日周四 下午10:45写道:
>
>> Hello,
>>
>> What is the correct way to use Python dict's as ROW type in pyflink? Im
>> trying this:
>>
>> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp' ],
>> [Types.STRING(), Types.STRING(), Types.LONG() ])
>>
>> class MyProcessFunction(KeyedProcessFunction):
>> def process_element(self, value, ctx: 'KeyedProcessFunction.Context', out: Collector):
>> result = {"id": ctx.get_current_key()[0], "data": "some_string", "timestamp": 111111111111}
>> out.collect(result)
>> current_watermark = ctx.timer_service().current_watermark()
>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>>
>> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext', out: 'Collector'):
>> logging.info(timestamp)
>> out.collect("On timer timestamp: " + str(timestamp))
>>
>> ds.key_by(MyKeySelector(), key_type_info=Types.TUPLE([Types.STRING(), Types.STRING()])) \
>> .process(MyProcessFunction(), output_type=output_type_info)
>>
>>
>> I just hardcoded the values in MyProcessFunction to be sure that the
>> input data doesnt mess the fields. So the data is correct but PyFlink trews
>> an exception:
>>
>> at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.MaskUtils.readIntoMask(MaskUtils.java:73)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:202)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>>> at
>>> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
>>> at
>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
>>> at
>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
>>> at
>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
>>> at
>>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
>>> ... 10 more
>>
>> However it works with primitive types like Types.STRING(). According to the documentation the ROW type corresponds to the python's dict type.
>>
>>
>> Regards
>>
>>
Re: Declaring and using ROW data type in PyFlink DataStream
Posted by Shuiqiang Chen <ac...@gmail.com>.
Hi meneldor,
The main cause of the error is that there is a bug in
`ctx.timer_service().current_watermark()`. At the beginning the stream,
when the first record come into the KeyedProcessFunction.process_element()
, the current_watermark will be the Long.MIN_VALUE at Java side, while at
the Python side, it becomes LONG.MAX_VALUE which is 9223372036854775807.
>>> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
Here, 9223372036854775807 + 1500 is 9223372036854777307 which will be
automatically converted to a long interger in python but will cause Long
value overflow in Java when deserializing the registered timer value. I
will craete a issue to fix the bug.
Let’s return to your initial question, at PyFlink you could create a Row
Type data as bellow:
>>> row_data = Row(id=‘my id’, data=’some data’, timestamp=1111)
And I wonder which release version of flink the code snippet you provided
based on? The latest API for KeyedProcessFunction.process_element() and
KeyedProcessFunction.on_timer() will not provid a `collector` to collect
output data but use `yield` which is a more pythonic approach.
Please refer to the following code:
def keyed_process_function_example():
env = StreamExecutionEnvironment.get_execution_environment()
env.set_parallelism(1)
env.get_config().set_auto_watermark_interval(2000)
env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
data_stream = env.from_collection([(1, 'hello', '1603708211000'),
(2, 'hi', '1603708224000'),
(3, 'hello', '1603708226000'),
(4, 'hi', '1603708289000')],
type_info=Types.ROW([Types.INT(), Types.STRING(), Types.STRING()]))
class MyTimestampAssigner(TimestampAssigner):
def extract_timestamp(self, value, record_timestamp) -> int:
return int(value[2])
class MyProcessFunction(KeyedProcessFunction):
def process_element(self, value, ctx: 'KeyedProcessFunction.Context'):
yield Row(id=ctx.get_current_key()[1], data='some_string',
timestamp=11111111)
# current_watermark = ctx.timer_service().current_watermark()
ctx.timer_service().register_event_time_timer(ctx.timestamp()
+ 1500)
def on_timer(self, timestamp: int, ctx:
'KeyedProcessFunction.OnTimerContext'):
yield Row(id=ctx.get_current_key()[1], data='current on
timer timestamp: ' + str(timestamp),
timestamp=timestamp)
output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp'],
[Types.STRING(), Types.STRING(), Types.INT()])
watermark_strategy = WatermarkStrategy.for_monotonous_timestamps() \
.with_timestamp_assigner(MyTimestampAssigner())
data_stream.assign_timestamps_and_watermarks(watermark_strategy) \
.key_by(lambda x: (x[0], x[1]),
key_type_info=Types.TUPLE([Types.INT(), Types.STRING()])) \
.process(MyProcessFunction(), output_type=output_type_info).print()
env.execute('test keyed process function')
Best,
Shuiqiang
meneldor <me...@gmail.com> 于2021年1月14日周四 下午10:45写道:
> Hello,
>
> What is the correct way to use Python dict's as ROW type in pyflink? Im
> trying this:
>
> output_type_info = Types.ROW_NAMED(['id', 'data', 'timestamp' ],
> [Types.STRING(), Types.STRING(), Types.LONG() ])
>
> class MyProcessFunction(KeyedProcessFunction):
> def process_element(self, value, ctx: 'KeyedProcessFunction.Context', out: Collector):
> result = {"id": ctx.get_current_key()[0], "data": "some_string", "timestamp": 111111111111}
> out.collect(result)
> current_watermark = ctx.timer_service().current_watermark()
> ctx.timer_service().register_event_time_timer(current_watermark + 1500)
>
> def on_timer(self, timestamp, ctx: 'KeyedProcessFunction.OnTimerContext', out: 'Collector'):
> logging.info(timestamp)
> out.collect("On timer timestamp: " + str(timestamp))
>
> ds.key_by(MyKeySelector(), key_type_info=Types.TUPLE([Types.STRING(), Types.STRING()])) \
> .process(MyProcessFunction(), output_type=output_type_info)
>
>
> I just hardcoded the values in MyProcessFunction to be sure that the input
> data doesnt mess the fields. So the data is correct but PyFlink trews an
> exception:
>
> at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
>> at
>> org.apache.flink.api.java.typeutils.runtime.MaskUtils.readIntoMask(MaskUtils.java:73)
>> at
>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:202)
>> at
>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>> at
>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:213)
>> at
>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.deserialize(RowSerializer.java:58)
>> at
>> org.apache.flink.streaming.api.operators.python.PythonKeyedProcessOperator.emitResult(PythonKeyedProcessOperator.java:253)
>> at
>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.emitResults(AbstractPythonFunctionOperator.java:266)
>> at
>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:293)
>> at
>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
>> at
>> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
>> ... 10 more
>
> However it works with primitive types like Types.STRING(). According to the documentation the ROW type corresponds to the python's dict type.
>
>
> Regards
>
>