You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Parag Somani <so...@gmail.com> on 2021/10/05 10:36:57 UTC
Issues while upgrading from 1.12.1 to 1.14.0
Hello,
We are currently using Apache flink 1.12.0 deployed on k8s cluster of 1.18
with zk for HA. Due to certain vulnerabilities in container related with
few jar(like netty-*, meso), we are forced to upgrade.
While upgrading flink to 1.14.0, faced NPE,
https://issues.apache.org/jira/browse/FLINK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17402570#comment-17402570
To address it, I have followed steps
1. savepoint creation
2. Stop the job
3. Restore from save point where i am facing challenge.
For step #3 from above, i was able to restore from savepoint mainly because:
"bin/flink run -s :savepointPath [:runArgs] "
It majorly about restarting a jar file uploaded. As our application is
based on k8s and running using docker, i was not able to restore it. And
because of it, state of variables in accumulator got corrupted and i lost
the data in one of env.
My query is, what is preffered way to restore from savepoint, if
application is running on k8s using docker.
We are using following command to run job manager:
/docker-entrypoint.sh "standalone-job" "-Ds3.access-key=
${AWS_ACCESS_KEY_ID}" "-Ds3.secret-key=${AWS_SECRET_ACCESS_KEY}"
"-Ds3.endpoint=${AWS_S3_ENDPOINT}" "-Dhigh-availability.zookeeper.quorum=
${ZOOKEEPER_CLUSTER}" "--job-classname" "<class-name>" ${args}
Thank you in advance...!
--
Regards,
Parag Surajmal Somani.
Re: Issues while upgrading from 1.12.1 to 1.14.0
Posted by Ali Bahadir Zeybek <al...@ververica.com>.
Hello Parag,
Looking at the last command you sent, it seems like you are not passing the
savepoint path for the savepoint instance, but just passing the savepoint
directory while restarting the job.
When a savepoint is completed, it is usually materialized under
<SAVEPOINT_DIRECTORY>/<JOB_ID>/savepoint-<SAVEPOINT_ID>.
Can you please try to find the latest successful savepoint under
<SAVEPOINT_DIRECTORY>/<JOB_ID> and pass the full path of the savepoint
instance?
Sincerely,
Ali
On Wed, Oct 6, 2021 at 12:58 PM Dawid Wysakowicz <dw...@apache.org>
wrote:
> Hi Parag,
>
> When you restore from a savepoint do you see a line like: "Restoring job
> {} from {}" in jobmanagers logs? Is the entire state lost or just part of
> it? Could you explain a bit what does your job look like and how do you
> check that the state is lost?
>
> Sorry if too obvious, but what are the "accumulators" you refer to? Are
> they *State primitives[1] or really constructs that are called
> Accumulator[2]? The latter are not checkpointed.
>
> Best,
>
> Dawid
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#using-keyed-state
>
> [2]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/user_defined_functions/#accumulators--counters
> On 06/10/2021 08:42, Parag Somani wrote:
>
> Yes Nico. I have evaluated this.
>
> I have tried below:
>
> 1. Take the savepoint
> 2. Stop the job
> 3. Shutdown the instances
> 4. Started new pod using below command:
>
> /docker-entrypoint.sh "standalone-job" "-Ds3.access-key=
> ${AWS_ACCESS_KEY_ID}" "-Ds3.secret-key=${AWS_SECRET_ACCESS_KEY}"
> "-Ds3.endpoint=${AWS_S3_ENDPOINT}" "-Dhigh-availability.zookeeper.quorum=
> ${ZOOKEEPER_CLUSTER}" "--job-classname" "com.test.MySpringBootApp"
> "--fromSavepoint" "s3://s3-health-service-discovery/savepoints" ${args}
>
> I haven't observed any errors during start-up in logs. But the state got
> reset i.e. values stored inside the accumulator got flushed.
>
> On Tue, Oct 5, 2021 at 9:40 PM Nicolaus Weidner <
> nicolaus.weidner@ververica.com> wrote:
>
>> Hi Parag,
>>
>> I am not so familiar with the setup you are using, but did you check out
>> [1]? Maybe the parameter
>> [--fromSavepoint /path/to/savepoint [--allowNonRestoredState]]
>> is what you are looking for?
>>
>> Best regards,
>> Nico
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#application-mode-on-docker
>>
>> On Tue, Oct 5, 2021 at 12:37 PM Parag Somani <so...@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> We are currently using Apache flink 1.12.0 deployed on k8s cluster of
>>> 1.18 with zk for HA. Due to certain vulnerabilities in container related
>>> with few jar(like netty-*, meso), we are forced to upgrade.
>>>
>>> While upgrading flink to 1.14.0, faced NPE,
>>> https://issues.apache.org/jira/browse/FLINK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17402570#comment-17402570
>>>
>>> To address it, I have followed steps
>>>
>>> 1. savepoint creation
>>> 2. Stop the job
>>> 3. Restore from save point where i am facing challenge.
>>>
>>> For step #3 from above, i was able to restore from savepoint mainly
>>> because:
>>> "bin/flink run -s :savepointPath [:runArgs] "
>>> It majorly about restarting a jar file uploaded. As our application is
>>> based on k8s and running using docker, i was not able to restore it. And
>>> because of it, state of variables in accumulator got corrupted and i lost
>>> the data in one of env.
>>>
>>> My query is, what is preffered way to restore from savepoint, if
>>> application is running on k8s using docker.
>>>
>>> We are using following command to run job manager:
>>> /docker-entrypoint.sh "standalone-job" "-Ds3.access-key=
>>> ${AWS_ACCESS_KEY_ID}" "-Ds3.secret-key=${AWS_SECRET_ACCESS_KEY}"
>>> "-Ds3.endpoint=${AWS_S3_ENDPOINT}"
>>> "-Dhigh-availability.zookeeper.quorum=${ZOOKEEPER_CLUSTER}"
>>> "--job-classname" "<class-name>" ${args}
>>>
>>> Thank you in advance...!
>>>
>>> --
>>> Regards,
>>> Parag Surajmal Somani.
>>>
>>
>
> --
> Regards,
> Parag Surajmal Somani.
>
>
--
Ali Bahadır Zeybek | Solutions Architect
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner
Re: Issues while upgrading from 1.12.1 to 1.14.0
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hi Parag,
When you restore from a savepoint do you see a line like: "Restoring job
{} from {}" in jobmanagers logs? Is the entire state lost or just part
of it? Could you explain a bit what does your job look like and how do
you check that the state is lost?
Sorry if too obvious, but what are the "accumulators" you refer to? Are
they *State primitives[1] or really constructs that are called
Accumulator[2]? The latter are not checkpointed.
Best,
Dawid
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#using-keyed-state
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/user_defined_functions/#accumulators--counters
On 06/10/2021 08:42, Parag Somani wrote:
> Yes Nico. I have evaluated this.
>
> I have tried below:
>
> 1. Take the savepoint
> 2. Stop the job
> 3. Shutdown the instances
> 4. Started new pod using below command:
>
> /docker-entrypoint.sh "standalone-job" "-Ds3.access-key=${AWS_ACCESS_KEY_ID}" "-Ds3.secret-key=${AWS_SECRET_ACCESS_KEY}" "-Ds3.endpoint=${AWS_S3_ENDPOINT}" "-Dhigh-availability.zookeeper.quorum=${ZOOKEEPER_CLUSTER}" "--job-classname" "com.test.MySpringBootApp" "--fromSavepoint" "s3://s3-health-service-discovery/savepoints" ${args}
>
> I haven't observed any errors during start-up in logs. But the state
> got reset i.e. values stored inside the accumulator got flushed.
>
> On Tue, Oct 5, 2021 at 9:40 PM Nicolaus Weidner
> <nicolaus.weidner@ververica.com
> <ma...@ververica.com>> wrote:
>
> Hi Parag,
>
> I am not so familiar with the setup you are using, but did you
> check out [1]? Maybe the parameter
> [--fromSavepoint /path/to/savepoint [--allowNonRestoredState]]
> is what you are looking for?
>
> Best regards,
> Nico
>
> [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#application-mode-on-docker
> <https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#application-mode-on-docker>
>
> On Tue, Oct 5, 2021 at 12:37 PM Parag Somani
> <somaniparag@gmail.com <ma...@gmail.com>> wrote:
>
> Hello,
>
> We are currently using Apache flink 1.12.0 deployed on k8s
> cluster of 1.18 with zk for HA. Due to certain vulnerabilities
> in container related with few jar(like netty-*, meso), we are
> forced to upgrade.
>
> While upgrading flink to 1.14.0, faced NPE,
> https://issues.apache.org/jira/browse/FLINK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17402570#comment-17402570
> <https://issues.apache.org/jira/browse/FLINK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17402570#comment-17402570>
>
> To address it, I have followed steps
>
> 1. savepoint creation
> 2. Stop the job
> 3. Restore from save point where i am facing challenge.
>
> For step #3 from above, i was able to restore from savepoint
> mainly because:
> "|bin/flink run -s :savepointPath [:runArgs]| "
> It majorly about restarting a jar file uploaded. As our
> application is based on k8s and running using docker, i was
> not able to restore it. And because of it, state of variables
> in accumulator got corrupted and i lost the data in one of env.
>
> My query is, what is preffered way to restore from savepoint,
> if application is running on k8s using docker.
>
> We are using following command to run job manager:
> /docker-entrypoint.sh "standalone-job" "-Ds3.access-key=${AWS_ACCESS_KEY_ID}" "-Ds3.secret-key=${AWS_SECRET_ACCESS_KEY}" "-Ds3.endpoint=${AWS_S3_ENDPOINT}" "-Dhigh-availability.zookeeper.quorum=${ZOOKEEPER_CLUSTER}" "--job-classname" "<class-name>" ${args}
>
> Thank you in advance...!
>
> --
> Regards,
> Parag Surajmal Somani.
>
>
>
> --
> Regards,
> Parag Surajmal Somani.
Re: Issues while upgrading from 1.12.1 to 1.14.0
Posted by Parag Somani <so...@gmail.com>.
Yes Nico. I have evaluated this.
I have tried below:
1. Take the savepoint
2. Stop the job
3. Shutdown the instances
4. Started new pod using below command:
/docker-entrypoint.sh "standalone-job" "-Ds3.access-key=${AWS_ACCESS_KEY_ID}
" "-Ds3.secret-key=${AWS_SECRET_ACCESS_KEY}" "-Ds3.endpoint=
${AWS_S3_ENDPOINT}" "-Dhigh-availability.zookeeper.quorum=
${ZOOKEEPER_CLUSTER}" "--job-classname" "com.test.MySpringBootApp"
"--fromSavepoint" "s3://s3-health-service-discovery/savepoints" ${args}
I haven't observed any errors during start-up in logs. But the state got
reset i.e. values stored inside the accumulator got flushed.
On Tue, Oct 5, 2021 at 9:40 PM Nicolaus Weidner <
nicolaus.weidner@ververica.com> wrote:
> Hi Parag,
>
> I am not so familiar with the setup you are using, but did you check out
> [1]? Maybe the parameter
> [--fromSavepoint /path/to/savepoint [--allowNonRestoredState]]
> is what you are looking for?
>
> Best regards,
> Nico
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#application-mode-on-docker
>
> On Tue, Oct 5, 2021 at 12:37 PM Parag Somani <so...@gmail.com>
> wrote:
>
>> Hello,
>>
>> We are currently using Apache flink 1.12.0 deployed on k8s cluster of
>> 1.18 with zk for HA. Due to certain vulnerabilities in container related
>> with few jar(like netty-*, meso), we are forced to upgrade.
>>
>> While upgrading flink to 1.14.0, faced NPE,
>> https://issues.apache.org/jira/browse/FLINK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17402570#comment-17402570
>>
>> To address it, I have followed steps
>>
>> 1. savepoint creation
>> 2. Stop the job
>> 3. Restore from save point where i am facing challenge.
>>
>> For step #3 from above, i was able to restore from savepoint mainly
>> because:
>> "bin/flink run -s :savepointPath [:runArgs] "
>> It majorly about restarting a jar file uploaded. As our application is
>> based on k8s and running using docker, i was not able to restore it. And
>> because of it, state of variables in accumulator got corrupted and i lost
>> the data in one of env.
>>
>> My query is, what is preffered way to restore from savepoint, if
>> application is running on k8s using docker.
>>
>> We are using following command to run job manager:
>> /docker-entrypoint.sh "standalone-job" "-Ds3.access-key=
>> ${AWS_ACCESS_KEY_ID}" "-Ds3.secret-key=${AWS_SECRET_ACCESS_KEY}"
>> "-Ds3.endpoint=${AWS_S3_ENDPOINT}" "-Dhigh-availability.zookeeper.quorum=
>> ${ZOOKEEPER_CLUSTER}" "--job-classname" "<class-name>" ${args}
>>
>> Thank you in advance...!
>>
>> --
>> Regards,
>> Parag Surajmal Somani.
>>
>
--
Regards,
Parag Surajmal Somani.
Re: Issues while upgrading from 1.12.1 to 1.14.0
Posted by Nicolaus Weidner <ni...@ververica.com>.
Hi Parag,
I am not so familiar with the setup you are using, but did you check out
[1]? Maybe the parameter
[--fromSavepoint /path/to/savepoint [--allowNonRestoredState]]
is what you are looking for?
Best regards,
Nico
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#application-mode-on-docker
On Tue, Oct 5, 2021 at 12:37 PM Parag Somani <so...@gmail.com> wrote:
> Hello,
>
> We are currently using Apache flink 1.12.0 deployed on k8s cluster of 1.18
> with zk for HA. Due to certain vulnerabilities in container related with
> few jar(like netty-*, meso), we are forced to upgrade.
>
> While upgrading flink to 1.14.0, faced NPE,
> https://issues.apache.org/jira/browse/FLINK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17402570#comment-17402570
>
> To address it, I have followed steps
>
> 1. savepoint creation
> 2. Stop the job
> 3. Restore from save point where i am facing challenge.
>
> For step #3 from above, i was able to restore from savepoint mainly
> because:
> "bin/flink run -s :savepointPath [:runArgs] "
> It majorly about restarting a jar file uploaded. As our application is
> based on k8s and running using docker, i was not able to restore it. And
> because of it, state of variables in accumulator got corrupted and i lost
> the data in one of env.
>
> My query is, what is preffered way to restore from savepoint, if
> application is running on k8s using docker.
>
> We are using following command to run job manager:
> /docker-entrypoint.sh "standalone-job" "-Ds3.access-key=
> ${AWS_ACCESS_KEY_ID}" "-Ds3.secret-key=${AWS_SECRET_ACCESS_KEY}"
> "-Ds3.endpoint=${AWS_S3_ENDPOINT}" "-Dhigh-availability.zookeeper.quorum=
> ${ZOOKEEPER_CLUSTER}" "--job-classname" "<class-name>" ${args}
>
> Thank you in advance...!
>
> --
> Regards,
> Parag Surajmal Somani.
>