You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by jingjing bai <ba...@gmail.com> on 2020/03/30 10:15:58 UTC
flink 1.10 support LONG as watermark?
Hi:
flinkers!
I try to upgrade our production to 1.10V from 1.9 which is our current
product version.
in our case,the event_time is Long ,and we had implement this function
which support long type as a watermark in our inner version, it is a
different from the official version on 1.10.
on 1.10 version, flink had add watermark definition and I had drop our
implement.
and i encounter this problem too ,
how can I do in new version, if is , I 'm not to migrate our inner
implement to new version.
Re: flink 1.10 support LONG as watermark?
Posted by Jingsong Li <ji...@gmail.com>.
Hi jingjing,
If seconds precision is OK for you.
You can try "to_timestamp(from_unixtime(your_time_seconds_long))".
Best,
Jingsong Lee
On Wed, Apr 1, 2020 at 8:56 AM jingjing bai <ba...@gmail.com>
wrote:
> Thanks a lot!
>
> Jark Wu <im...@gmail.com> 于2020年4月1日周三 上午1:13写道：
>
>> Hi Jing,
>>
>> I created https://issues.apache.org/jira/browse/FLINK-16889 to support
>> converting from BIGINT to TIMESTAMP.
>>
>> Best,
>> Jark
>>
>> On Mon, 30 Mar 2020 at 20:30, jingjing bai <ba...@gmail.com>
>> wrote:
>>
>>> Hi jarkWu!
>>>
>>> Is there a FLIP to do so? I'm very glad to learn from idea.
>>>
>>>
>>> Best,
>>> jing
>>>
>>> Jark Wu <im...@gmail.com> 于2020年3月30日周一 下午6:52写道：
>>>
>>>> Hi Jingjing,
>>>>
>>>> Event time field must be a TIMESTAMP(3) type. You can convert your Long
>>>> type value into TIMESTAMP(3) using user-defined function.
>>>> I'm sorry that Flink doesn't provide built-in function for this
>>>> purpose, but will have one soon.
>>>>
>>>> For example:
>>>> CREATE TABLE myTable (
>>>> log_ts bigint,
>>>> event_time AS my_func(log_ts),
>>>> WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND
>>>> ) WITH (
>>>> ...
>>>> );
>>>>
>>>> Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).
>>>>
>>>> Best,
>>>> Jark
>>>>
>>>> On Mon, 30 Mar 2020 at 18:16, jingjing bai <ba...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Hi:
>>>>> flinkers!
>>>>>
>>>>> I try to upgrade our production to 1.10V from 1.9 which is our current
>>>>> product version.
>>>>> in our case,the event_time is Long ,and we had implement this function
>>>>> which support long type as a watermark in our inner version, it is a
>>>>> different from the official version on 1.10.
>>>>> on 1.10 version, flink had add watermark definition and I had drop our
>>>>> implement.
>>>>> and i encounter this problem too ,
>>>>>
>>>>> how can I do in new version, if is , I 'm not to migrate our inner
>>>>> implement to new version.
>>>>>
>>>>>
>>>>>
--
Best, Jingsong Lee
Re: flink 1.10 support LONG as watermark?
Posted by jingjing bai <ba...@gmail.com>.
Thanks a lot!
Jark Wu <im...@gmail.com> 于2020年4月1日周三 上午1:13写道：
> Hi Jing,
>
> I created https://issues.apache.org/jira/browse/FLINK-16889 to support
> converting from BIGINT to TIMESTAMP.
>
> Best,
> Jark
>
> On Mon, 30 Mar 2020 at 20:30, jingjing bai <ba...@gmail.com>
> wrote:
>
>> Hi jarkWu!
>>
>> Is there a FLIP to do so? I'm very glad to learn from idea.
>>
>>
>> Best,
>> jing
>>
>> Jark Wu <im...@gmail.com> 于2020年3月30日周一 下午6:52写道：
>>
>>> Hi Jingjing,
>>>
>>> Event time field must be a TIMESTAMP(3) type. You can convert your Long
>>> type value into TIMESTAMP(3) using user-defined function.
>>> I'm sorry that Flink doesn't provide built-in function for this purpose,
>>> but will have one soon.
>>>
>>> For example:
>>> CREATE TABLE myTable (
>>> log_ts bigint,
>>> event_time AS my_func(log_ts),
>>> WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND
>>> ) WITH (
>>> ...
>>> );
>>>
>>> Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).
>>>
>>> Best,
>>> Jark
>>>
>>> On Mon, 30 Mar 2020 at 18:16, jingjing bai <ba...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Hi:
>>>> flinkers!
>>>>
>>>> I try to upgrade our production to 1.10V from 1.9 which is our current
>>>> product version.
>>>> in our case,the event_time is Long ,and we had implement this function
>>>> which support long type as a watermark in our inner version, it is a
>>>> different from the official version on 1.10.
>>>> on 1.10 version, flink had add watermark definition and I had drop our
>>>> implement.
>>>> and i encounter this problem too ,
>>>>
>>>> how can I do in new version, if is , I 'm not to migrate our inner
>>>> implement to new version.
>>>>
>>>>
>>>>
Re: flink 1.10 support LONG as watermark?
Posted by Jark Wu <im...@gmail.com>.
Hi Jing,
I created https://issues.apache.org/jira/browse/FLINK-16889 to support
converting from BIGINT to TIMESTAMP.
Best,
Jark
On Mon, 30 Mar 2020 at 20:30, jingjing bai <ba...@gmail.com>
wrote:
> Hi jarkWu!
>
> Is there a FLIP to do so? I'm very glad to learn from idea.
>
>
> Best,
> jing
>
> Jark Wu <im...@gmail.com> 于2020年3月30日周一 下午6:52写道：
>
>> Hi Jingjing,
>>
>> Event time field must be a TIMESTAMP(3) type. You can convert your Long
>> type value into TIMESTAMP(3) using user-defined function.
>> I'm sorry that Flink doesn't provide built-in function for this purpose,
>> but will have one soon.
>>
>> For example:
>> CREATE TABLE myTable (
>> log_ts bigint,
>> event_time AS my_func(log_ts),
>> WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND
>> ) WITH (
>> ...
>> );
>>
>> Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).
>>
>> Best,
>> Jark
>>
>> On Mon, 30 Mar 2020 at 18:16, jingjing bai <ba...@gmail.com>
>> wrote:
>>
>>>
>>> Hi:
>>> flinkers!
>>>
>>> I try to upgrade our production to 1.10V from 1.9 which is our current
>>> product version.
>>> in our case,the event_time is Long ,and we had implement this function
>>> which support long type as a watermark in our inner version, it is a
>>> different from the official version on 1.10.
>>> on 1.10 version, flink had add watermark definition and I had drop our
>>> implement.
>>> and i encounter this problem too ,
>>>
>>> how can I do in new version, if is , I 'm not to migrate our inner
>>> implement to new version.
>>>
>>>
>>>
Re: flink 1.10 support LONG as watermark?
Posted by jingjing bai <ba...@gmail.com>.
Hi jarkWu!
Is there a FLIP to do so? I'm very glad to learn from idea.
Best,
jing
Jark Wu <im...@gmail.com> 于2020年3月30日周一 下午6:52写道：
> Hi Jingjing,
>
> Event time field must be a TIMESTAMP(3) type. You can convert your Long
> type value into TIMESTAMP(3) using user-defined function.
> I'm sorry that Flink doesn't provide built-in function for this purpose,
> but will have one soon.
>
> For example:
> CREATE TABLE myTable (
> log_ts bigint,
> event_time AS my_func(log_ts),
> WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND
> ) WITH (
> ...
> );
>
> Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).
>
> Best,
> Jark
>
> On Mon, 30 Mar 2020 at 18:16, jingjing bai <ba...@gmail.com>
> wrote:
>
>>
>> Hi:
>> flinkers!
>>
>> I try to upgrade our production to 1.10V from 1.9 which is our current
>> product version.
>> in our case,the event_time is Long ,and we had implement this function
>> which support long type as a watermark in our inner version, it is a
>> different from the official version on 1.10.
>> on 1.10 version, flink had add watermark definition and I had drop our
>> implement.
>> and i encounter this problem too ,
>>
>> how can I do in new version, if is , I 'm not to migrate our inner
>> implement to new version.
>>
>>
>>
Re: flink 1.10 support LONG as watermark?
Posted by Jark Wu <im...@gmail.com>.
Hi Jingjing,
Event time field must be a TIMESTAMP(3) type. You can convert your Long
type value into TIMESTAMP(3) using user-defined function.
I'm sorry that Flink doesn't provide built-in function for this purpose,
but will have one soon.
For example:
CREATE TABLE myTable (
log_ts bigint,
event_time AS my_func(log_ts),
WATERMARK FOR event_time AS event_time - INTERVAL '1' SECOND
) WITH (
...
);
Here my_func is a UDF which converts BIGINT into TIMESTAMP(3).
Best,
Jark
On Mon, 30 Mar 2020 at 18:16, jingjing bai <ba...@gmail.com>
wrote:
>
> Hi:
> flinkers!
>
> I try to upgrade our production to 1.10V from 1.9 which is our current
> product version.
> in our case,the event_time is Long ,and we had implement this function
> which support long type as a watermark in our inner version, it is a
> different from the official version on 1.10.
> on 1.10 version, flink had add watermark definition and I had drop our
> implement.
> and i encounter this problem too ,
>
> how can I do in new version, if is , I 'm not to migrate our inner
> implement to new version.
>
>
>