You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishal Sharma <vi...@grab.com> on 2019/06/19 14:26:45 UTC
[External] checkpoint metadata not in configured directory state.checkpoints.dir
Hi Folks,
I am using flink 1.8 with externalised checkpointing enabled and saving the
checkpoints to aws S3.
My configuration is as follows :
flink-conf.yaml :
state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata
In application code :
env.setStateBackend(new
RocksDBStateBackend("s3a://test-bucket/checkpoints", true))
As per my understanding, the externalized checkpoint’s meta data is
determined from the configuration key "state.checkpoints.dir" and
checkpoint data is stored in state backend path.
However, In my case, I don't see anything in the metadata directory. The
_metadata file is present inside each of the checkpoint directory (chk-6043
...).
Is this the expected behavior ? If yes, what is the use of
"state.checkpoints.dir" configuration ?
My goal is to establish a process to automatically restart the job from
last completed externalised checkpoint in case of failure. For this
to happen, I need to able to figure out path for the metadata of latest
checkpoint.
Thanks,
Vishal Sharma
--
*_Grab is hiring. Learn more at _**https://grab.careers
<https://grab.careers/>*
By communicating with Grab Inc and/or its
subsidiaries, associate companies and jointly controlled entities (“Grab
Group”), you are deemed to have consented to the processing of your
personal data as set out in the Privacy Notice which can be viewed at
https://grab.com/privacy/ <https://grab.com/privacy/>
This email contains
confidential information and is only for the intended recipient(s). If you
are not the intended recipient(s), please do not disseminate, distribute or
copy this email Please notify Grab Group immediately if you have received
this by mistake and delete this email from your system. Email transmission
cannot be guaranteed to be secure or error-free as any information therein
could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or
contain viruses. Grab Group do not accept liability for any errors or
omissions in the contents of this email arises as a result of email
transmission. All intellectual property rights in this email and
attachments therein shall remain vested in Grab Group, unless otherwise
provided by law.
Re: [External] checkpoint metadata not in configured directory state.checkpoints.dir
Posted by Congxian Qiu <qc...@gmail.com>.
Hi Vishal
Maybe the slide[1] (page 40) can be helpful
[1] https://files.alicdn.com/tpsservice/c421720fcb1c51026257cd770923844a.pdf
Best,
Congxian
Vishal Sharma <vi...@grab.com> 于2019年6月20日周四 下午5:42写道：
> Hi Congxian,
>
> I am not sure how can I track the checkpoint path. Can you suggestion
> regarding this ?
>
> Thanks,
> Vishal Sharma
>
> On Thu, Jun 20, 2019 at 11:17 AM Congxian Qiu <qc...@gmail.com>
> wrote:
>
>> Hi, Vishal
>> If you want to restart from the last competed external checkpoint of the
>> previous stoped job, you need to track the checkpoint path and restart from
>> it.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint
>>
>> Best,
>> Congxian
>>
>>
>> Vishal Sharma <vi...@grab.com> 于2019年6月19日周三 下午11:38写道：
>>
>>> Hi Chesnay,
>>>
>>> Can you suggest, How should I go about automating job restart from last
>>> completed externalised checkpoint in case of failure ? I am not sure about
>>> the path for the latest completed checkpoint.
>>>
>>> Thanks,
>>> Vishal Sharma
>>>
>>> On Wed, Jun 19, 2019 at 11:11 PM Chesnay Schepler <ch...@apache.org>
>>> wrote:
>>>
>>>> The _metadata is always stored in the same directory as the checkpoint
>>>> data.
>>>>
>>>> As outlined here
>>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/checkpoints.html#directory-structure>
>>>> "state.checkpoints.dir" serves as a cluster-wide configuration that _can_
>>>> be overwritten with a job-specific setting when creating the state-backend.
>>>>
>>>> If you want the state-backend to use the configured directory you must
>>>> configure the state-backend in the configuration as well, as outlined
>>>> here
>>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#setting-default-state-backend>
>>>> .
>>>>
>>>> On 19/06/2019 16:26, Vishal Sharma wrote:
>>>>
>>>> Hi Folks,
>>>>
>>>> I am using flink 1.8 with externalised checkpointing enabled and saving
>>>> the checkpoints to aws S3.
>>>>
>>>> My configuration is as follows :
>>>>
>>>> flink-conf.yaml :
>>>> state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata
>>>>
>>>> In application code :
>>>> env.setStateBackend(new
>>>> RocksDBStateBackend("s3a://test-bucket/checkpoints", true))
>>>>
>>>> As per my understanding, the externalized checkpoint’s meta data is
>>>> determined from the configuration key "state.checkpoints.dir" and
>>>> checkpoint data is stored in state backend path.
>>>>
>>>> However, In my case, I don't see anything in the metadata directory.
>>>> The _metadata file is present inside each of the checkpoint directory (chk-6043
>>>> ...).
>>>>
>>>> Is this the expected behavior ? If yes, what is the use of
>>>> "state.checkpoints.dir" configuration ?
>>>>
>>>> My goal is to establish a process to automatically restart the job from
>>>> last completed externalised checkpoint in case of failure. For this
>>>> to happen, I need to able to figure out path for the metadata of latest
>>>> checkpoint.
>>>>
>>>> Thanks,
>>>> Vishal Sharma
>>>>
>>>> *Grab is hiring. Learn more at https://grab.careers
>>>> <https://grab.careers/>*
>>>>
>>>> By communicating with Grab Inc and/or its subsidiaries, associate
>>>> companies and jointly controlled entities (“Grab Group”), you are deemed to
>>>> have consented to the processing of your personal data as set out in the
>>>> Privacy Notice which can be viewed at https://grab.com/privacy/
>>>>
>>>> This email contains confidential information and is only for the
>>>> intended recipient(s). If you are not the intended recipient(s), please do
>>>> not disseminate, distribute or copy this email Please notify Grab Group
>>>> immediately if you have received this by mistake and delete this email from
>>>> your system. Email transmission cannot be guaranteed to be secure or
>>>> error-free as any information therein could be intercepted, corrupted,
>>>> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
>>>> not accept liability for any errors or omissions in the contents of this
>>>> email arises as a result of email transmission. All intellectual property
>>>> rights in this email and attachments therein shall remain vested in Grab
>>>> Group, unless otherwise provided by law.
>>>>
>>>>
>>>>
>>> *Grab is hiring. Learn more at https://grab.careers
>>> <https://grab.careers/>*
>>>
>>> By communicating with Grab Inc and/or its subsidiaries, associate
>>> companies and jointly controlled entities (“Grab Group”), you are deemed to
>>> have consented to the processing of your personal data as set out in the
>>> Privacy Notice which can be viewed at https://grab.com/privacy/
>>>
>>> This email contains confidential information and is only for the
>>> intended recipient(s). If you are not the intended recipient(s), please do
>>> not disseminate, distribute or copy this email Please notify Grab Group
>>> immediately if you have received this by mistake and delete this email from
>>> your system. Email transmission cannot be guaranteed to be secure or
>>> error-free as any information therein could be intercepted, corrupted,
>>> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
>>> not accept liability for any errors or omissions in the contents of this
>>> email arises as a result of email transmission. All intellectual property
>>> rights in this email and attachments therein shall remain vested in Grab
>>> Group, unless otherwise provided by law.
>>>
>>
> *Grab is hiring. Learn more at https://grab.careers
> <https://grab.careers/>*
>
> By communicating with Grab Inc and/or its subsidiaries, associate
> companies and jointly controlled entities (“Grab Group”), you are deemed to
> have consented to the processing of your personal data as set out in the
> Privacy Notice which can be viewed at https://grab.com/privacy/
>
> This email contains confidential information and is only for the intended
> recipient(s). If you are not the intended recipient(s), please do not
> disseminate, distribute or copy this email Please notify Grab Group
> immediately if you have received this by mistake and delete this email from
> your system. Email transmission cannot be guaranteed to be secure or
> error-free as any information therein could be intercepted, corrupted,
> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
> not accept liability for any errors or omissions in the contents of this
> email arises as a result of email transmission. All intellectual property
> rights in this email and attachments therein shall remain vested in Grab
> Group, unless otherwise provided by law.
>
Re: [External] checkpoint metadata not in configured directory state.checkpoints.dir
Posted by Vishal Sharma <vi...@grab.com>.
Hi Congxian,
I am not sure how can I track the checkpoint path. Can you suggestion
regarding this ?
Thanks,
Vishal Sharma
On Thu, Jun 20, 2019 at 11:17 AM Congxian Qiu <qc...@gmail.com>
wrote:
> Hi, Vishal
> If you want to restart from the last competed external checkpoint of the
> previous stoped job, you need to track the checkpoint path and restart from
> it.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint
>
> Best,
> Congxian
>
>
> Vishal Sharma <vi...@grab.com> 于2019年6月19日周三 下午11:38写道：
>
>> Hi Chesnay,
>>
>> Can you suggest, How should I go about automating job restart from last
>> completed externalised checkpoint in case of failure ? I am not sure about
>> the path for the latest completed checkpoint.
>>
>> Thanks,
>> Vishal Sharma
>>
>> On Wed, Jun 19, 2019 at 11:11 PM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> The _metadata is always stored in the same directory as the checkpoint
>>> data.
>>>
>>> As outlined here
>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/checkpoints.html#directory-structure>
>>> "state.checkpoints.dir" serves as a cluster-wide configuration that _can_
>>> be overwritten with a job-specific setting when creating the state-backend.
>>>
>>> If you want the state-backend to use the configured directory you must
>>> configure the state-backend in the configuration as well, as outlined
>>> here
>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#setting-default-state-backend>
>>> .
>>>
>>> On 19/06/2019 16:26, Vishal Sharma wrote:
>>>
>>> Hi Folks,
>>>
>>> I am using flink 1.8 with externalised checkpointing enabled and saving
>>> the checkpoints to aws S3.
>>>
>>> My configuration is as follows :
>>>
>>> flink-conf.yaml :
>>> state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata
>>>
>>> In application code :
>>> env.setStateBackend(new
>>> RocksDBStateBackend("s3a://test-bucket/checkpoints", true))
>>>
>>> As per my understanding, the externalized checkpoint’s meta data is
>>> determined from the configuration key "state.checkpoints.dir" and
>>> checkpoint data is stored in state backend path.
>>>
>>> However, In my case, I don't see anything in the metadata directory. The
>>> _metadata file is present inside each of the checkpoint directory (chk-6043
>>> ...).
>>>
>>> Is this the expected behavior ? If yes, what is the use of
>>> "state.checkpoints.dir" configuration ?
>>>
>>> My goal is to establish a process to automatically restart the job from
>>> last completed externalised checkpoint in case of failure. For this
>>> to happen, I need to able to figure out path for the metadata of latest
>>> checkpoint.
>>>
>>> Thanks,
>>> Vishal Sharma
>>>
>>> *Grab is hiring. Learn more at https://grab.careers
>>> <https://grab.careers/>*
>>>
>>> By communicating with Grab Inc and/or its subsidiaries, associate
>>> companies and jointly controlled entities (“Grab Group”), you are deemed to
>>> have consented to the processing of your personal data as set out in the
>>> Privacy Notice which can be viewed at https://grab.com/privacy/
>>>
>>> This email contains confidential information and is only for the
>>> intended recipient(s). If you are not the intended recipient(s), please do
>>> not disseminate, distribute or copy this email Please notify Grab Group
>>> immediately if you have received this by mistake and delete this email from
>>> your system. Email transmission cannot be guaranteed to be secure or
>>> error-free as any information therein could be intercepted, corrupted,
>>> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
>>> not accept liability for any errors or omissions in the contents of this
>>> email arises as a result of email transmission. All intellectual property
>>> rights in this email and attachments therein shall remain vested in Grab
>>> Group, unless otherwise provided by law.
>>>
>>>
>>>
>> *Grab is hiring. Learn more at https://grab.careers
>> <https://grab.careers/>*
>>
>> By communicating with Grab Inc and/or its subsidiaries, associate
>> companies and jointly controlled entities (“Grab Group”), you are deemed to
>> have consented to the processing of your personal data as set out in the
>> Privacy Notice which can be viewed at https://grab.com/privacy/
>>
>> This email contains confidential information and is only for the intended
>> recipient(s). If you are not the intended recipient(s), please do not
>> disseminate, distribute or copy this email Please notify Grab Group
>> immediately if you have received this by mistake and delete this email from
>> your system. Email transmission cannot be guaranteed to be secure or
>> error-free as any information therein could be intercepted, corrupted,
>> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
>> not accept liability for any errors or omissions in the contents of this
>> email arises as a result of email transmission. All intellectual property
>> rights in this email and attachments therein shall remain vested in Grab
>> Group, unless otherwise provided by law.
>>
>
--
*_Grab is hiring. Learn more at _**https://grab.careers
<https://grab.careers/>*
By communicating with Grab Inc and/or its
subsidiaries, associate companies and jointly controlled entities (“Grab
Group”), you are deemed to have consented to the processing of your
personal data as set out in the Privacy Notice which can be viewed at
https://grab.com/privacy/ <https://grab.com/privacy/>
This email contains
confidential information and is only for the intended recipient(s). If you
are not the intended recipient(s), please do not disseminate, distribute or
copy this email Please notify Grab Group immediately if you have received
this by mistake and delete this email from your system. Email transmission
cannot be guaranteed to be secure or error-free as any information therein
could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or
contain viruses. Grab Group do not accept liability for any errors or
omissions in the contents of this email arises as a result of email
transmission. All intellectual property rights in this email and
attachments therein shall remain vested in Grab Group, unless otherwise
provided by law.
Re: [External] checkpoint metadata not in configured directory state.checkpoints.dir
Posted by Congxian Qiu <qc...@gmail.com>.
Hi, Vishal
If you want to restart from the last competed external checkpoint of the
previous stoped job, you need to track the checkpoint path and restart from
it.
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint
Best,
Congxian
Vishal Sharma <vi...@grab.com> 于2019年6月19日周三 下午11:38写道：
> Hi Chesnay,
>
> Can you suggest, How should I go about automating job restart from last
> completed externalised checkpoint in case of failure ? I am not sure about
> the path for the latest completed checkpoint.
>
> Thanks,
> Vishal Sharma
>
> On Wed, Jun 19, 2019 at 11:11 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> The _metadata is always stored in the same directory as the checkpoint
>> data.
>>
>> As outlined here
>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/checkpoints.html#directory-structure>
>> "state.checkpoints.dir" serves as a cluster-wide configuration that _can_
>> be overwritten with a job-specific setting when creating the state-backend.
>>
>> If you want the state-backend to use the configured directory you must
>> configure the state-backend in the configuration as well, as outlined
>> here
>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#setting-default-state-backend>
>> .
>>
>> On 19/06/2019 16:26, Vishal Sharma wrote:
>>
>> Hi Folks,
>>
>> I am using flink 1.8 with externalised checkpointing enabled and saving
>> the checkpoints to aws S3.
>>
>> My configuration is as follows :
>>
>> flink-conf.yaml :
>> state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata
>>
>> In application code :
>> env.setStateBackend(new
>> RocksDBStateBackend("s3a://test-bucket/checkpoints", true))
>>
>> As per my understanding, the externalized checkpoint’s meta data is
>> determined from the configuration key "state.checkpoints.dir" and
>> checkpoint data is stored in state backend path.
>>
>> However, In my case, I don't see anything in the metadata directory. The
>> _metadata file is present inside each of the checkpoint directory (chk-6043
>> ...).
>>
>> Is this the expected behavior ? If yes, what is the use of
>> "state.checkpoints.dir" configuration ?
>>
>> My goal is to establish a process to automatically restart the job from
>> last completed externalised checkpoint in case of failure. For this
>> to happen, I need to able to figure out path for the metadata of latest
>> checkpoint.
>>
>> Thanks,
>> Vishal Sharma
>>
>> *Grab is hiring. Learn more at https://grab.careers
>> <https://grab.careers/>*
>>
>> By communicating with Grab Inc and/or its subsidiaries, associate
>> companies and jointly controlled entities (“Grab Group”), you are deemed to
>> have consented to the processing of your personal data as set out in the
>> Privacy Notice which can be viewed at https://grab.com/privacy/
>>
>> This email contains confidential information and is only for the intended
>> recipient(s). If you are not the intended recipient(s), please do not
>> disseminate, distribute or copy this email Please notify Grab Group
>> immediately if you have received this by mistake and delete this email from
>> your system. Email transmission cannot be guaranteed to be secure or
>> error-free as any information therein could be intercepted, corrupted,
>> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
>> not accept liability for any errors or omissions in the contents of this
>> email arises as a result of email transmission. All intellectual property
>> rights in this email and attachments therein shall remain vested in Grab
>> Group, unless otherwise provided by law.
>>
>>
>>
> *Grab is hiring. Learn more at https://grab.careers
> <https://grab.careers/>*
>
> By communicating with Grab Inc and/or its subsidiaries, associate
> companies and jointly controlled entities (“Grab Group”), you are deemed to
> have consented to the processing of your personal data as set out in the
> Privacy Notice which can be viewed at https://grab.com/privacy/
>
> This email contains confidential information and is only for the intended
> recipient(s). If you are not the intended recipient(s), please do not
> disseminate, distribute or copy this email Please notify Grab Group
> immediately if you have received this by mistake and delete this email from
> your system. Email transmission cannot be guaranteed to be secure or
> error-free as any information therein could be intercepted, corrupted,
> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
> not accept liability for any errors or omissions in the contents of this
> email arises as a result of email transmission. All intellectual property
> rights in this email and attachments therein shall remain vested in Grab
> Group, unless otherwise provided by law.
>
Re: [External] checkpoint metadata not in configured directory state.checkpoints.dir
Posted by Vishal Sharma <vi...@grab.com>.
Hi Chesnay,
Can you suggest, How should I go about automating job restart from last
completed externalised checkpoint in case of failure ? I am not sure about
the path for the latest completed checkpoint.
Thanks,
Vishal Sharma
On Wed, Jun 19, 2019 at 11:11 PM Chesnay Schepler <ch...@apache.org>
wrote:
> The _metadata is always stored in the same directory as the checkpoint
> data.
>
> As outlined here
> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/checkpoints.html#directory-structure>
> "state.checkpoints.dir" serves as a cluster-wide configuration that _can_
> be overwritten with a job-specific setting when creating the state-backend.
>
> If you want the state-backend to use the configured directory you must
> configure the state-backend in the configuration as well, as outlined here
> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#setting-default-state-backend>
> .
>
> On 19/06/2019 16:26, Vishal Sharma wrote:
>
> Hi Folks,
>
> I am using flink 1.8 with externalised checkpointing enabled and saving
> the checkpoints to aws S3.
>
> My configuration is as follows :
>
> flink-conf.yaml :
> state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata
>
> In application code :
> env.setStateBackend(new
> RocksDBStateBackend("s3a://test-bucket/checkpoints", true))
>
> As per my understanding, the externalized checkpoint’s meta data is
> determined from the configuration key "state.checkpoints.dir" and
> checkpoint data is stored in state backend path.
>
> However, In my case, I don't see anything in the metadata directory. The
> _metadata file is present inside each of the checkpoint directory (chk-6043
> ...).
>
> Is this the expected behavior ? If yes, what is the use of
> "state.checkpoints.dir" configuration ?
>
> My goal is to establish a process to automatically restart the job from
> last completed externalised checkpoint in case of failure. For this
> to happen, I need to able to figure out path for the metadata of latest
> checkpoint.
>
> Thanks,
> Vishal Sharma
>
> *Grab is hiring. Learn more at https://grab.careers
> <https://grab.careers/>*
>
> By communicating with Grab Inc and/or its subsidiaries, associate
> companies and jointly controlled entities (“Grab Group”), you are deemed to
> have consented to the processing of your personal data as set out in the
> Privacy Notice which can be viewed at https://grab.com/privacy/
>
> This email contains confidential information and is only for the intended
> recipient(s). If you are not the intended recipient(s), please do not
> disseminate, distribute or copy this email Please notify Grab Group
> immediately if you have received this by mistake and delete this email from
> your system. Email transmission cannot be guaranteed to be secure or
> error-free as any information therein could be intercepted, corrupted,
> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
> not accept liability for any errors or omissions in the contents of this
> email arises as a result of email transmission. All intellectual property
> rights in this email and attachments therein shall remain vested in Grab
> Group, unless otherwise provided by law.
>
>
>
--
*_Grab is hiring. Learn more at _**https://grab.careers
<https://grab.careers/>*
By communicating with Grab Inc and/or its
subsidiaries, associate companies and jointly controlled entities (“Grab
Group”), you are deemed to have consented to the processing of your
personal data as set out in the Privacy Notice which can be viewed at
https://grab.com/privacy/ <https://grab.com/privacy/>
This email contains
confidential information and is only for the intended recipient(s). If you
are not the intended recipient(s), please do not disseminate, distribute or
copy this email Please notify Grab Group immediately if you have received
this by mistake and delete this email from your system. Email transmission
cannot be guaranteed to be secure or error-free as any information therein
could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or
contain viruses. Grab Group do not accept liability for any errors or
omissions in the contents of this email arises as a result of email
transmission. All intellectual property rights in this email and
attachments therein shall remain vested in Grab Group, unless otherwise
provided by law.
Re: [External] checkpoint metadata not in configured directory
state.checkpoints.dir
Posted by Chesnay Schepler <ch...@apache.org>.
The _metadata is always stored in the same directory as the checkpoint data.
As outlined here
<https://ci.apache.org/projects/flink/flink-docs-master/ops/state/checkpoints.html#directory-structure>
"state.checkpoints.dir" serves as a cluster-wide configuration that
_can_ be overwritten with a job-specific setting when creating the
state-backend.
If you want the state-backend to use the configured directory you must
configure the state-backend in the configuration as well, as outlined
here
<https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#setting-default-state-backend>.
On 19/06/2019 16:26, Vishal Sharma wrote:
> Hi Folks,
>
> I am using flink 1.8 with externalised checkpointing enabled and
> saving the checkpoints to aws S3.
>
> My configuration is as follows :
>
> flink-conf.yaml :
> state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata
>
> In application code :
> env.setStateBackend(new
> RocksDBStateBackend("s3a://test-bucket/checkpoints", true))
>
> As per my understanding, the externalized checkpoint’s meta data is
> determined from the configuration key "|state.checkpoints.dir" and
> checkpoint data is stored in state backend path. |
> |
> |
> |However, In my case, I don't see anything in the metadata directory.
> The _metadata file is present inside each of the checkpoint directory
> (chk-6043 ...).
> |
> |
> |
> Is this the expected behavior ? If yes, what is the use of
> "state.checkpoints.dir" configuration ?
>
> My goal is to establish a process to automatically restart the job
> from last completed externalised checkpoint in case of failure. For
> this to happen, I need to able to figure out path for the metadata of
> latest checkpoint.
>
> Thanks,
> Vishal Sharma
>
> */Grab is hiring. Learn more at //https://grab.careers
> <https://grab.careers/>/*
>
> By communicating with Grab Inc and/or its subsidiaries, associate
> companies and jointly controlled entities (“Grab Group”), you are
> deemed to have consented to the processing of your personal data as
> set out in the Privacy Notice which can be viewed at
> https://grab.com/privacy/
>
> This email contains confidential information and is only for the
> intended recipient(s). If you are not the intended recipient(s),
> please do not disseminate, distribute or copy this email Please notify
> Grab Group immediately if you have received this by mistake and delete
> this email from your system. Email transmission cannot be guaranteed
> to be secure or error-free as any information therein could be
> intercepted, corrupted, lost, destroyed, delayed or incomplete, or
> contain viruses. Grab Group do not accept liability for any errors or
> omissions in the contents of this email arises as a result of email
> transmission. All intellectual property rights in this email and
> attachments therein shall remain vested in Grab Group, unless
> otherwise provided by law.