You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by LakeShen <sh...@gmail.com> on 2020/03/13 03:30:55 UTC
Cancel the flink task and restore from checkpoint ,can I change the
flink operator's parallelism
Hi community,
I have a question is that I cancel the flink task and retain the
checkpoint dir, then restore from the checkpoint dir ,can I change the
flink operator's parallelism,in my thoughts, I think I can't change the
flink operator's parallelism,but I am not sure.
Thanks to your reply.
Best wishes,
LakeShen
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by LakeShen <sh...@gmail.com>.
Thank you, I will do that.
jinhai wang <ji...@gmail.com> 于2020年3月17日周二 下午5:58写道：
> Hi LakeShen
>
> You also must assign IDs to all operators of an application. Otherwise,
> you may not be able to recover from checkpoint
>
> Doc:
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state
>
>
> Best Regards
>
> jinhai.me@gmail.com
>
> 2020年3月17日 下午5:51，Till Rohrmann <tr...@apache.org> 写道：
>
> Let us know if you should run into any problems. The state processor API
> is still young and could benefit from as much feedback as possible.
>
> Cheers,
> Till
>
> On Tue, Mar 17, 2020 at 2:57 AM LakeShen <sh...@gmail.com>
> wrote:
>
>> Wow,this feature is look so good,I will see this feature.
>> Thank you to reply me , Till😀.
>>
>> Best Regards,
>> LakeShen
>>
>> Till Rohrmann <tr...@apache.org> 于2020年3月17日周二 上午4:17写道：
>>
>>> If you want to change the max parallelism then you need to take a
>>> savepoint and use Flink's state processor API [1] to rewrite the max
>>> parallelism by creating a new savepoint from the old one.
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>
>>> Cheers,
>>> Till
>>>
>>> On Sat, Mar 14, 2020 at 4:07 AM LakeShen <sh...@gmail.com>
>>> wrote:
>>>
>>>> Hi Eleanore , if you resume from savepoint , you can't change the flink
>>>> operator's max parallelism .
>>>>
>>>> Eleanore Jin <el...@gmail.com> 于2020年3月14日周六 上午12:51写道：
>>>>
>>>> > Hi Piotr,
>>>> > Does this also apply to savepoint? (meaning the max parallelism
>>>> should not
>>>> > change for job resume from savepoint?)
>>>> >
>>>> > Thanks a lot!
>>>> > Eleanore
>>>> >
>>>> > On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <pi...@ververica.com>
>>>> > wrote:
>>>> >
>>>> > > Hi,
>>>> > >
>>>> > > Yes, you can change the parallelism. One thing that you can not
>>>> change is
>>>> > > “max parallelism”.
>>>> > >
>>>> > > Piotrek
>>>> > >
>>>> > > > On 13 Mar 2020, at 04:34, Sivaprasanna <sivaprasanna246@gmail.com
>>>> >
>>>> > > wrote:
>>>> > > >
>>>> > > > I think you can modify the operator’s parallelism. It is only if
>>>> you
>>>> > > have set maxParallelism, and while restoring from a checkpoint, you
>>>> > > shouldn’t modify the maxParallelism. Otherwise, I believe the state
>>>> will
>>>> > be
>>>> > > lost.
>>>> > > >
>>>> > > > -
>>>> > > > Sivaprasanna
>>>> > > >
>>>> > > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <
>>>> shenleifighting@gmail.com
>>>> > > <ma...@gmail.com>> wrote:
>>>> > > > Hi community,
>>>> > > > I have a question is that I cancel the flink task and
>>>> retain the
>>>> > > checkpoint dir, then restore from the checkpoint dir ,can I change
>>>> the
>>>> > > flink operator's parallelism,in my thoughts, I think I can't change
>>>> the
>>>> > > flink operator's parallelism,but I am not sure.
>>>> > > > Thanks to your reply.
>>>> > > >
>>>> > > > Best wishes,
>>>> > > > LakeShen
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
>
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by jinhai wang <ji...@gmail.com>.
Hi LakeShen
You also must assign IDs to all operators of an application. Otherwise, you may not be able to recover from checkpoint
Doc: https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state <https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state>
Best Regards
jinhai.me@gmail.com
> 2020年3月17日 下午5:51，Till Rohrmann <tr...@apache.org> 写道：
>
> Let us know if you should run into any problems. The state processor API is still young and could benefit from as much feedback as possible.
>
> Cheers,
> Till
>
> On Tue, Mar 17, 2020 at 2:57 AM LakeShen <shenleifighting@gmail.com <ma...@gmail.com>> wrote:
> Wow,this feature is look so good,I will see this feature.
> Thank you to reply me , Till😀.
>
> Best Regards,
> LakeShen
>
> Till Rohrmann <trohrmann@apache.org <ma...@apache.org>> 于2020年3月17日周二 上午4:17写道：
> If you want to change the max parallelism then you need to take a savepoint and use Flink's state processor API [1] to rewrite the max parallelism by creating a new savepoint from the old one.
>
> [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html <https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html>
>
> Cheers,
> Till
>
> On Sat, Mar 14, 2020 at 4:07 AM LakeShen <shenleifighting@gmail.com <ma...@gmail.com>> wrote:
> Hi Eleanore , if you resume from savepoint , you can't change the flink
> operator's max parallelism .
>
> Eleanore Jin <eleanore.jin@gmail.com <ma...@gmail.com>> 于2020年3月14日周六 上午12:51写道：
>
> > Hi Piotr,
> > Does this also apply to savepoint? (meaning the max parallelism should not
> > change for job resume from savepoint?)
> >
> > Thanks a lot!
> > Eleanore
> >
> > On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <piotr@ververica.com <ma...@ververica.com>>
> > wrote:
> >
> > > Hi,
> > >
> > > Yes, you can change the parallelism. One thing that you can not change is
> > > “max parallelism”.
> > >
> > > Piotrek
> > >
> > > > On 13 Mar 2020, at 04:34, Sivaprasanna <sivaprasanna246@gmail.com <ma...@gmail.com>>
> > > wrote:
> > > >
> > > > I think you can modify the operator’s parallelism. It is only if you
> > > have set maxParallelism, and while restoring from a checkpoint, you
> > > shouldn’t modify the maxParallelism. Otherwise, I believe the state will
> > be
> > > lost.
> > > >
> > > > -
> > > > Sivaprasanna
> > > >
> > > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <shenleifighting@gmail.com <ma...@gmail.com>
> > > <mailto:shenleifighting@gmail.com <ma...@gmail.com>>> wrote:
> > > > Hi community,
> > > > I have a question is that I cancel the flink task and retain the
> > > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > > flink operator's parallelism,in my thoughts, I think I can't change the
> > > flink operator's parallelism,but I am not sure.
> > > > Thanks to your reply.
> > > >
> > > > Best wishes,
> > > > LakeShen
> > >
> > >
> >
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by Till Rohrmann <tr...@apache.org>.
Let us know if you should run into any problems. The state processor API is
still young and could benefit from as much feedback as possible.
Cheers,
Till
On Tue, Mar 17, 2020 at 2:57 AM LakeShen <sh...@gmail.com> wrote:
> Wow,this feature is look so good,I will see this feature.
> Thank you to reply me , Till😀.
>
> Best Regards,
> LakeShen
>
> Till Rohrmann <tr...@apache.org> 于2020年3月17日周二 上午4:17写道：
>
>> If you want to change the max parallelism then you need to take a
>> savepoint and use Flink's state processor API [1] to rewrite the max
>> parallelism by creating a new savepoint from the old one.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>
>> Cheers,
>> Till
>>
>> On Sat, Mar 14, 2020 at 4:07 AM LakeShen <sh...@gmail.com>
>> wrote:
>>
>>> Hi Eleanore , if you resume from savepoint , you can't change the flink
>>> operator's max parallelism .
>>>
>>> Eleanore Jin <el...@gmail.com> 于2020年3月14日周六 上午12:51写道：
>>>
>>> > Hi Piotr,
>>> > Does this also apply to savepoint? (meaning the max parallelism should
>>> not
>>> > change for job resume from savepoint?)
>>> >
>>> > Thanks a lot!
>>> > Eleanore
>>> >
>>> > On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <pi...@ververica.com>
>>> > wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > Yes, you can change the parallelism. One thing that you can not
>>> change is
>>> > > “max parallelism”.
>>> > >
>>> > > Piotrek
>>> > >
>>> > > > On 13 Mar 2020, at 04:34, Sivaprasanna <si...@gmail.com>
>>> > > wrote:
>>> > > >
>>> > > > I think you can modify the operator’s parallelism. It is only if
>>> you
>>> > > have set maxParallelism, and while restoring from a checkpoint, you
>>> > > shouldn’t modify the maxParallelism. Otherwise, I believe the state
>>> will
>>> > be
>>> > > lost.
>>> > > >
>>> > > > -
>>> > > > Sivaprasanna
>>> > > >
>>> > > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <
>>> shenleifighting@gmail.com
>>> > > <ma...@gmail.com>> wrote:
>>> > > > Hi community,
>>> > > > I have a question is that I cancel the flink task and retain
>>> the
>>> > > checkpoint dir, then restore from the checkpoint dir ,can I change
>>> the
>>> > > flink operator's parallelism,in my thoughts, I think I can't change
>>> the
>>> > > flink operator's parallelism,but I am not sure.
>>> > > > Thanks to your reply.
>>> > > >
>>> > > > Best wishes,
>>> > > > LakeShen
>>> > >
>>> > >
>>> >
>>>
>>
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by Till Rohrmann <tr...@apache.org>.
If you want to change the max parallelism then you need to take a savepoint
and use Flink's state processor API [1] to rewrite the max parallelism by
creating a new savepoint from the old one.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
Cheers,
Till
On Sat, Mar 14, 2020 at 4:07 AM LakeShen <sh...@gmail.com> wrote:
> Hi Eleanore , if you resume from savepoint , you can't change the flink
> operator's max parallelism .
>
> Eleanore Jin <el...@gmail.com> 于2020年3月14日周六 上午12:51写道：
>
> > Hi Piotr,
> > Does this also apply to savepoint? (meaning the max parallelism should
> not
> > change for job resume from savepoint?)
> >
> > Thanks a lot!
> > Eleanore
> >
> > On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <pi...@ververica.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Yes, you can change the parallelism. One thing that you can not change
> is
> > > “max parallelism”.
> > >
> > > Piotrek
> > >
> > > > On 13 Mar 2020, at 04:34, Sivaprasanna <si...@gmail.com>
> > > wrote:
> > > >
> > > > I think you can modify the operator’s parallelism. It is only if you
> > > have set maxParallelism, and while restoring from a checkpoint, you
> > > shouldn’t modify the maxParallelism. Otherwise, I believe the state
> will
> > be
> > > lost.
> > > >
> > > > -
> > > > Sivaprasanna
> > > >
> > > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <shenleifighting@gmail.com
> > > <ma...@gmail.com>> wrote:
> > > > Hi community,
> > > > I have a question is that I cancel the flink task and retain
> the
> > > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > > flink operator's parallelism,in my thoughts, I think I can't change the
> > > flink operator's parallelism,but I am not sure.
> > > > Thanks to your reply.
> > > >
> > > > Best wishes,
> > > > LakeShen
> > >
> > >
> >
>
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by Till Rohrmann <tr...@apache.org>.
If you want to change the max parallelism then you need to take a savepoint
and use Flink's state processor API [1] to rewrite the max parallelism by
creating a new savepoint from the old one.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
Cheers,
Till
On Sat, Mar 14, 2020 at 4:07 AM LakeShen <sh...@gmail.com> wrote:
> Hi Eleanore , if you resume from savepoint , you can't change the flink
> operator's max parallelism .
>
> Eleanore Jin <el...@gmail.com> 于2020年3月14日周六 上午12:51写道：
>
> > Hi Piotr,
> > Does this also apply to savepoint? (meaning the max parallelism should
> not
> > change for job resume from savepoint?)
> >
> > Thanks a lot!
> > Eleanore
> >
> > On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <pi...@ververica.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Yes, you can change the parallelism. One thing that you can not change
> is
> > > “max parallelism”.
> > >
> > > Piotrek
> > >
> > > > On 13 Mar 2020, at 04:34, Sivaprasanna <si...@gmail.com>
> > > wrote:
> > > >
> > > > I think you can modify the operator’s parallelism. It is only if you
> > > have set maxParallelism, and while restoring from a checkpoint, you
> > > shouldn’t modify the maxParallelism. Otherwise, I believe the state
> will
> > be
> > > lost.
> > > >
> > > > -
> > > > Sivaprasanna
> > > >
> > > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <shenleifighting@gmail.com
> > > <ma...@gmail.com>> wrote:
> > > > Hi community,
> > > > I have a question is that I cancel the flink task and retain
> the
> > > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > > flink operator's parallelism,in my thoughts, I think I can't change the
> > > flink operator's parallelism,but I am not sure.
> > > > Thanks to your reply.
> > > >
> > > > Best wishes,
> > > > LakeShen
> > >
> > >
> >
>
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by LakeShen <sh...@gmail.com>.
Hi Eleanore , if you resume from savepoint , you can't change the flink
operator's max parallelism .
Eleanore Jin <el...@gmail.com> 于2020年3月14日周六 上午12:51写道：
> Hi Piotr,
> Does this also apply to savepoint? (meaning the max parallelism should not
> change for job resume from savepoint?)
>
> Thanks a lot!
> Eleanore
>
> On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <pi...@ververica.com>
> wrote:
>
> > Hi,
> >
> > Yes, you can change the parallelism. One thing that you can not change is
> > “max parallelism”.
> >
> > Piotrek
> >
> > > On 13 Mar 2020, at 04:34, Sivaprasanna <si...@gmail.com>
> > wrote:
> > >
> > > I think you can modify the operator’s parallelism. It is only if you
> > have set maxParallelism, and while restoring from a checkpoint, you
> > shouldn’t modify the maxParallelism. Otherwise, I believe the state will
> be
> > lost.
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <shenleifighting@gmail.com
> > <ma...@gmail.com>> wrote:
> > > Hi community,
> > > I have a question is that I cancel the flink task and retain the
> > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > flink operator's parallelism,in my thoughts, I think I can't change the
> > flink operator's parallelism,but I am not sure.
> > > Thanks to your reply.
> > >
> > > Best wishes,
> > > LakeShen
> >
> >
>
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by Eleanore Jin <el...@gmail.com>.
Hi Piotr,
Does this also apply to savepoint? (meaning the max parallelism should not
change for job resume from savepoint?)
Thanks a lot!
Eleanore
On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <pi...@ververica.com> wrote:
> Hi,
>
> Yes, you can change the parallelism. One thing that you can not change is
> “max parallelism”.
>
> Piotrek
>
> > On 13 Mar 2020, at 04:34, Sivaprasanna <si...@gmail.com>
> wrote:
> >
> > I think you can modify the operator’s parallelism. It is only if you
> have set maxParallelism, and while restoring from a checkpoint, you
> shouldn’t modify the maxParallelism. Otherwise, I believe the state will be
> lost.
> >
> > -
> > Sivaprasanna
> >
> > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <shenleifighting@gmail.com
> <ma...@gmail.com>> wrote:
> > Hi community,
> > I have a question is that I cancel the flink task and retain the
> checkpoint dir, then restore from the checkpoint dir ,can I change the
> flink operator's parallelism,in my thoughts, I think I can't change the
> flink operator's parallelism,but I am not sure.
> > Thanks to your reply.
> >
> > Best wishes,
> > LakeShen
>
>
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by Piotr Nowojski <pi...@ververica.com>.
Hi,
Yes, you can change the parallelism. One thing that you can not change is “max parallelism”.
Piotrek
> On 13 Mar 2020, at 04:34, Sivaprasanna <si...@gmail.com> wrote:
>
> I think you can modify the operator’s parallelism. It is only if you have set maxParallelism, and while restoring from a checkpoint, you shouldn’t modify the maxParallelism. Otherwise, I believe the state will be lost.
>
> -
> Sivaprasanna
>
> On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <shenleifighting@gmail.com <ma...@gmail.com>> wrote:
> Hi community,
> I have a question is that I cancel the flink task and retain the checkpoint dir, then restore from the checkpoint dir ,can I change the flink operator's parallelism,in my thoughts, I think I can't change the flink operator's parallelism,but I am not sure.
> Thanks to your reply.
>
> Best wishes,
> LakeShen
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by Piotr Nowojski <pi...@ververica.com>.
Hi,
Yes, you can change the parallelism. One thing that you can not change is “max parallelism”.
Piotrek
> On 13 Mar 2020, at 04:34, Sivaprasanna <si...@gmail.com> wrote:
>
> I think you can modify the operator’s parallelism. It is only if you have set maxParallelism, and while restoring from a checkpoint, you shouldn’t modify the maxParallelism. Otherwise, I believe the state will be lost.
>
> -
> Sivaprasanna
>
> On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <shenleifighting@gmail.com <ma...@gmail.com>> wrote:
> Hi community,
> I have a question is that I cancel the flink task and retain the checkpoint dir, then restore from the checkpoint dir ,can I change the flink operator's parallelism,in my thoughts, I think I can't change the flink operator's parallelism,but I am not sure.
> Thanks to your reply.
>
> Best wishes,
> LakeShen
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by Sivaprasanna <si...@gmail.com>.
I think you can modify the operator’s parallelism. It is only if you have
set maxParallelism, and while restoring from a checkpoint, you shouldn’t
modify the maxParallelism. Otherwise, I believe the state will be lost.
-
Sivaprasanna
On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <sh...@gmail.com> wrote:
> Hi community,
> I have a question is that I cancel the flink task and retain the
> checkpoint dir, then restore from the checkpoint dir ,can I change the
> flink operator's parallelism,in my thoughts, I think I can't change the
> flink operator's parallelism,but I am not sure.
> Thanks to your reply.
>
> Best wishes,
> LakeShen
>
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by Sivaprasanna <si...@gmail.com>.
I think you can modify the operator’s parallelism. It is only if you have
set maxParallelism, and while restoring from a checkpoint, you shouldn’t
modify the maxParallelism. Otherwise, I believe the state will be lost.
-
Sivaprasanna
On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <sh...@gmail.com> wrote:
> Hi community,
> I have a question is that I cancel the flink task and retain the
> checkpoint dir, then restore from the checkpoint dir ,can I change the
> flink operator's parallelism,in my thoughts, I think I can't change the
> flink operator's parallelism,but I am not sure.
> Thanks to your reply.
>
> Best wishes,
> LakeShen
>
Re: Cancel the flink task and restore from checkpoint ,can I change
the flink operator's parallelism
Posted by Sivaprasanna <si...@gmail.com>.
I think you can modify the operator’s parallelism. It is only if you have
set maxParallelism, and while restoring from a checkpoint, you shouldn’t
modify the maxParallelism. Otherwise, I believe the state will be lost.
-
Sivaprasanna
On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <sh...@gmail.com> wrote:
> Hi community,
> I have a question is that I cancel the flink task and retain the
> checkpoint dir, then restore from the checkpoint dir ,can I change the
> flink operator's parallelism,in my thoughts, I think I can't change the
> flink operator's parallelism,but I am not sure.
> Thanks to your reply.
>
> Best wishes,
> LakeShen
>