You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by leo65535 <le...@163.com> on 2021/12/22 10:14:06 UTC
[DISCUSS] Upgrade flink version
Hi, everyone
Description
SeaTunnel project use flink1.9 which is too old version for production, it's the time to upgrade the flink version.
But we need to consider, flink1.13 and flink1.9 are not compatible, so if we upgrade flink version directly,
we will maybe lost flink engine support for a long time, and we will rewrite all flink plugins one by one.
In addition, it will be very hard to upgrade flink version later, and the Flink API has good compatibility from flink.1.12 version.
And, if we want to write to lakehouse project(hudi, iceberg), we must use at least flink.1.12 version.
We need to think out a better way to solve it, looking for your ideas, thanks.
best,
leo65535
Re:Re:Re:[DISCUSS] Upgrade flink version
Posted by leo65535 <le...@163.com>.
Hi, developers
In this stage, we should not move forward recklessly, it's better to choose flink-1.13.5 as the new version.
The develop api of flink1.13 has better compatibility, it contains old and new api,
this will help us upgrade the version. And flink cdc[1] requires 1.13 version, it can't work with flink.1.14.
Ok, here my detail steps to upgrade the version.
Step1: Upgrade the flink to 1.13, ensure the original components are available as much as possible.
Step2: In order to let sql and conf job share the same connectors, we need to add new job conf parser, and new core api
Step3: Rich the connectors based on new core api.
[1] https://github.com/ververica/flink-cdc-connectors
Best wishes,
Leo65535
At 2021-12-23 15:50:11, "leo65535" <le...@163.com> wrote:
>
>
>Hi,
>
>
>I have an idea, we can release a SeaTunnel 2.0.4 version first, then we can start to work on upgrade flink version.
>If so, we will not break any users.
>
>
>Best wishes,
>Leo65535
>
>
>
>
>At 2021-12-22 18:47:47, "CalvinKirs" <ac...@163.com> wrote:
>>The other is how many users we have in the historical version of 2.X.
>>
>>
>>Best wishes！
>>Calvin Kirs
>>
>>
>>On 12/22/2021 18:44，CalvinKirs<ac...@163.com> wrote：
>>For me, now may be the best time, but I'm not sure how many users of 1.9 have.
>>
>>
>>Best wishes！
>>Calvin Kirs
>>
>>
>>On 12/22/2021 18:14，leo65535<le...@163.com> wrote：
>>
>>
>>Hi, everyone
>>
>>
>>Description
>>SeaTunnel project use flink1.9 which is too old version for production, it's the time to upgrade the flink version.
>>But we need to consider, flink1.13 and flink1.9 are not compatible, so if we upgrade flink version directly,
>>we will maybe lost flink engine support for a long time, and we will rewrite all flink plugins one by one.
>>
>>
>>In addition, it will be very hard to upgrade flink version later, and the Flink API has good compatibility from flink.1.12 version.
>>And, if we want to write to lakehouse project(hudi, iceberg), we must use at least flink.1.12 version.
>>
>>
>>We need to think out a better way to solve it, looking for your ideas, thanks.
>>
>>
>>best,
>>leo65535
>>
>>
>>
>>
Re:Re: [DISCUSS] Upgrade flink version
Posted by leo65535 <le...@163.com>.
Yeah, this is a big change, need more developers are needed to participate in the discussion.
Best,
Leo65535
在 2021-12-23 16:02:49，"zhuangchong" <zh...@163.com> 写道：
>Hi:
>
>You can release a current seatunel version first, but it is not recommended to continue to add components
>The engine version upgrade should be promoted first, and more plugins will be expanded later.
>
>
>
>> 2021年12月23日 下午3:50，leo65535 <le...@163.com> 写道：
>>
>> I have an idea, we can release a SeaTunnel 2.0.4 version first
>
Re: [DISCUSS] Upgrade flink version
Posted by zhuangchong <zh...@163.com>.
Hi:
You can release a current seatunel version first, but it is not recommended to continue to add components
The engine version upgrade should be promoted first, and more plugins will be expanded later.
> 2021年12月23日 下午3:50，leo65535 <le...@163.com> 写道：
>
> I have an idea, we can release a SeaTunnel 2.0.4 version first
Re:Re:[DISCUSS] Upgrade flink version
Posted by leo65535 <le...@163.com>.
Hi,
I have an idea, we can release a SeaTunnel 2.0.4 version first, then we can start to work on upgrade flink version.
If so, we will not break any users.
Best wishes,
Leo65535
At 2021-12-22 18:47:47, "CalvinKirs" <ac...@163.com> wrote:
>The other is how many users we have in the historical version of 2.X.
>
>
>Best wishes！
>Calvin Kirs
>
>
>On 12/22/2021 18:44，CalvinKirs<ac...@163.com> wrote：
>For me, now may be the best time, but I'm not sure how many users of 1.9 have.
>
>
>Best wishes！
>Calvin Kirs
>
>
>On 12/22/2021 18:14，leo65535<le...@163.com> wrote：
>
>
>Hi, everyone
>
>
>Description
>SeaTunnel project use flink1.9 which is too old version for production, it's the time to upgrade the flink version.
>But we need to consider, flink1.13 and flink1.9 are not compatible, so if we upgrade flink version directly,
>we will maybe lost flink engine support for a long time, and we will rewrite all flink plugins one by one.
>
>
>In addition, it will be very hard to upgrade flink version later, and the Flink API has good compatibility from flink.1.12 version.
>And, if we want to write to lakehouse project(hudi, iceberg), we must use at least flink.1.12 version.
>
>
>We need to think out a better way to solve it, looking for your ideas, thanks.
>
>
>best,
>leo65535
>
>
>
>
Re:[DISCUSS] Upgrade flink version
Posted by CalvinKirs <ac...@163.com>.
The other is how many users we have in the historical version of 2.X.
Best wishes！
Calvin Kirs
On 12/22/2021 18:44，CalvinKirs<ac...@163.com> wrote：
For me, now may be the best time, but I'm not sure how many users of 1.9 have.
Best wishes！
Calvin Kirs
On 12/22/2021 18:14，leo65535<le...@163.com> wrote：
Hi, everyone
Description
SeaTunnel project use flink1.9 which is too old version for production, it's the time to upgrade the flink version.
But we need to consider, flink1.13 and flink1.9 are not compatible, so if we upgrade flink version directly,
we will maybe lost flink engine support for a long time, and we will rewrite all flink plugins one by one.
In addition, it will be very hard to upgrade flink version later, and the Flink API has good compatibility from flink.1.12 version.
And, if we want to write to lakehouse project(hudi, iceberg), we must use at least flink.1.12 version.
We need to think out a better way to solve it, looking for your ideas, thanks.
best,
leo65535
Re:[DISCUSS] Upgrade flink version
Posted by CalvinKirs <ac...@163.com>.
For me, now may be the best time, but I'm not sure how many users of 1.9 have.
Best wishes！
Calvin Kirs
On 12/22/2021 18:14，leo65535<le...@163.com> wrote：
Hi, everyone
Description
SeaTunnel project use flink1.9 which is too old version for production, it's the time to upgrade the flink version.
But we need to consider, flink1.13 and flink1.9 are not compatible, so if we upgrade flink version directly,
we will maybe lost flink engine support for a long time, and we will rewrite all flink plugins one by one.
In addition, it will be very hard to upgrade flink version later, and the Flink API has good compatibility from flink.1.12 version.
And, if we want to write to lakehouse project(hudi, iceberg), we must use at least flink.1.12 version.
We need to think out a better way to solve it, looking for your ideas, thanks.
best,
leo65535