You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Juan Gentile <j....@criteo.com> on 2018/04/25 13:07:47 UTC
Externalized checkpoints and metadata
Hello,
We are trying to use externalized checkpoints, using RocksDB on Hadoop hdfs.
We would like to know what is the proper way to resume from a saved checkpoint as we are currently running many jobs in the same flink cluster.
The problem is that when we want to restart the jobs and pass the metadata file (or directory) there is 1 file per job but they are not easily identifiable based on the name:
Example
/checkpoints/checkpoint_metadata-69053704a5ca
/checkpoints/checkpoint_metadata-c7c016909607
We are not using savepoints and reading the documentation I see there are 2 ways to resume, 1 passing the metadata file (not possible as we have many jobs) and the other passing the directory,
But by default it looks for a _metadata file which doesn’t exist.
Thank you,
Juan G.
Re: Externalized checkpoints and metadata
Posted by 周思华 <su...@163.com>.
Hi Juan,
I think you are right and there maybe more then 3 companies implementing different solutions for this...I created a ticket to address it here https://issues.apache.org/jira/browse/FLINK-9260. Hope this could help to reduce other's redundant efforts on this...(If it could be accepted by community finally)
Best Regards,
Sihua Zhou
On 04/26/2018 16:35，Juan Gentile<j....@criteo.com> wrote：
Hello all,
Thank you all for your responses, I’ll take a look at your code Hao, and probably implement something similar.
I’d like to ask though, so as to know what we could expect from Flink in the future, if this issue will be addressed somehow, considering that we have already 3 different companies implementing different (but similar) solutions to solve the same problem.
Maybe we could think of adding this issue to here: https://cwiki.apache.org/confluence/display/FLINK/FLIP-10%3A+Unify+Checkpoints+and+Savepoints ?
Thank you,
Juan G.
From: hao gao <ha...@gmail.com>
Date: Wednesday, 25 April 2018 at 20:25
To: Juan Gentile <j....@criteo.com>
Cc: "user@flink.apache.org" <us...@flink.apache.org>, Oleksandr Nitavskyi <o....@criteo.com>
Subject: Re: Externalized checkpoints and metadata
Hi Juan,
We modified the flink code a little bit to change the flink checkpoint structure so we can easily identify which is which
you can read my note or the PR
https://medium.com/hadoop-noob/flink-externalized-checkpoint-eb86e693cfed
https://github.com/BranchMetrics/flink/pull/6/files
Hope it helps
Thanks
Hao
2018-04-25 6:07 GMT-07:00 Juan Gentile <j....@criteo.com>:
Hello,
We are trying to use externalized checkpoints, using RocksDB on Hadoop hdfs.
We would like to know what is the proper way to resume from a saved checkpoint as we are currently running many jobs in the same flink cluster.
The problem is that when we want to restart the jobs and pass the metadata file (or directory) there is 1 file per job but they are not easily identifiable based on the name:
Example
/checkpoints/checkpoint_metadata-69053704a5ca
/checkpoints/checkpoint_metadata-c7c016909607
We are not using savepoints and reading the documentation I see there are 2 ways to resume, 1 passing the metadata file (not possible as we have many jobs) and the other passing the directory,
But by default it looks for a _metadata file which doesn’t exist.
Thank you,
Juan G.
--
Thanks
- Hao
Re: Externalized checkpoints and metadata
Posted by Juan Gentile <j....@criteo.com>.
Hello all,
Thank you all for your responses, I’ll take a look at your code Hao, and probably implement something similar.
I’d like to ask though, so as to know what we could expect from Flink in the future, if this issue will be addressed somehow, considering that we have already 3 different companies implementing different (but similar) solutions to solve the same problem.
Maybe we could think of adding this issue to here: https://cwiki.apache.org/confluence/display/FLINK/FLIP-10%3A+Unify+Checkpoints+and+Savepoints ?
Thank you,
Juan G.
From: hao gao <ha...@gmail.com>
Date: Wednesday, 25 April 2018 at 20:25
To: Juan Gentile <j....@criteo.com>
Cc: "user@flink.apache.org" <us...@flink.apache.org>, Oleksandr Nitavskyi <o....@criteo.com>
Subject: Re: Externalized checkpoints and metadata
Hi Juan,
We modified the flink code a little bit to change the flink checkpoint structure so we can easily identify which is which
you can read my note or the PR
https://medium.com/hadoop-noob/flink-externalized-checkpoint-eb86e693cfed<https://urldefense.proofpoint.com/v2/url?u=https-3A__medium.com_hadoop-2Dnoob_flink-2Dexternalized-2Dcheckpoint-2Deb86e693cfed&d=DwMFaQ&c=nxfEpP1JWHVKAq835DW4mA&r=z5BFHEFwsu2ghSzcXn1_8T3-VzeesIO2aULbUy2urus&m=wx0Ifg9GgdI4lnIayewodxapkGzFntRi49BAsQ9KkTI&s=ZEmQ0XemTJ7FvyxIxmJ0psQbEXaYBE44wNdWPGuW9xY&e=>
https://github.com/BranchMetrics/flink/pull/6/files<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_BranchMetrics_flink_pull_6_files&d=DwMFaQ&c=nxfEpP1JWHVKAq835DW4mA&r=z5BFHEFwsu2ghSzcXn1_8T3-VzeesIO2aULbUy2urus&m=wx0Ifg9GgdI4lnIayewodxapkGzFntRi49BAsQ9KkTI&s=7PuWmJLmePMwtME15_Dp0O9W6FMe_ENwop5A9m42eO0&e=>
Hope it helps
Thanks
Hao
2018-04-25 6:07 GMT-07:00 Juan Gentile <j....@criteo.com>>:
Hello,
We are trying to use externalized checkpoints, using RocksDB on Hadoop hdfs.
We would like to know what is the proper way to resume from a saved checkpoint as we are currently running many jobs in the same flink cluster.
The problem is that when we want to restart the jobs and pass the metadata file (or directory) there is 1 file per job but they are not easily identifiable based on the name:
Example
/checkpoints/checkpoint_metadata-69053704a5ca
/checkpoints/checkpoint_metadata-c7c016909607
We are not using savepoints and reading the documentation I see there are 2 ways to resume, 1 passing the metadata file (not possible as we have many jobs) and the other passing the directory,
But by default it looks for a _metadata file which doesn’t exist.
Thank you,
Juan G.
--
Thanks
- Hao
Re: Externalized checkpoints and metadata
Posted by hao gao <ha...@gmail.com>.
Hi Juan,
We modified the flink code a little bit to change the flink checkpoint
structure so we can easily identify which is which
you can read my note or the PR
https://medium.com/hadoop-noob/flink-externalized-checkpoint-eb86e693cfed
https://github.com/BranchMetrics/flink/pull/6/files
Hope it helps
Thanks
Hao
2018-04-25 6:07 GMT-07:00 Juan Gentile <j....@criteo.com>:
> Hello,
>
>
>
> We are trying to use externalized checkpoints, using RocksDB on Hadoop
> hdfs.
>
> We would like to know what is the proper way to resume from a saved
> checkpoint as we are currently running many jobs in the same flink cluster.
>
>
>
> The problem is that when we want to restart the jobs and pass the metadata
> file (or directory) there is 1 file per job but they are not easily
> identifiable based on the name:
>
> Example
>
> */checkpoints/checkpoint_metadata-69053704a5ca*
>
> */checkpoints/checkpoint_metadata-c7c016909607*
>
>
>
> We are not using savepoints and reading the documentation I see there are
> 2 ways to resume, 1 passing the metadata file (not possible as we have many
> jobs) and the other passing the directory,
>
> But by default it looks for a _metadata file which doesn’t exist.
>
>
>
> Thank you,
>
> Juan G.
>
--
Thanks
- Hao