You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Abdullah bin Omar <ab...@gmail.com> on 2021/04/22 14:54:12 UTC
Question about snapshot file
Hi,
(1) what 's the snapshot metadata file (binary) contains ? is it possible
to read the snapshot metadata file by using Flink Deserialization?
(2) is there any function that can be used to see the previous states on
time of operation?
Thank you
Re: Question about snapshot file
Posted by Abdullah bin Omar <ab...@gmail.com>.
Thank you so much for your reply.
I apologise I did not mention multiple savepoint files in my last question.
I understand the part. I did not ask the question (only for one savepoint
file) exactly. When we run a job, we have obviously many savepoint files
(by using a manual command repeatedly)
I am asking: is it possible to extract all savepoint files data?
Thank you again
On Fri, Apr 30, 2021 at 12:42 PM Abdullah bin Omar <
abdullahbinomar29@gmail.com> wrote:
> Thank you so much for your reply.
>
> I apologise I did not mention multiple savepoint files in my last
> question.
>
> I understand the part. I did not ask the question (only for one savepoint
> file) exactly. When we run a job, we have obviously many savepoint files
> (by using a manual command repeatedly)
>
> I am asking: is it possible to extract all savepoint files data?
>
> Thank you again
>
> On Fri, Apr 30, 2021 at 12:01 PM David Anderson <da...@apache.org>
> wrote:
>
>> So, can't we extract all previous savepoint data by using
>>> ExistingSavepoint?
>>
>>
>> You can extract all of the data from any specific savepoint. Or nearly
>> all data, anyway. There is at least one corner case that isn't covered --
>> ListCheckpointed state -- which has been deprecated and isn't supported by
>> the savepoint API.
>>
>> David
>>
>> On Fri, Apr 30, 2021 at 5:42 PM Abdullah bin Omar <
>> abdullahbinomar29@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> So, can't we extract all previous savepoint data by using
>>> ExistingSavepoint?
>>>
>>>
>>> Thank you
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Apr 30, 2021 at 10:25 AM David Anderson <da...@apache.org>
>>> wrote:
>>>
>>>> Abdullah,
>>>>
>>>> The example you are studying -- the one using the state processor API
>>>> -- can be used with any retained checkpoint or savepoint created while
>>>> running the RidesAndFaresSolution job. But this is a very special use of
>>>> checkpoints and savepoints that shows how to extract data from them.
>>>>
>>>> Normally the state processor API is used with savepoints, and not with
>>>> checkpoints. This example uses checkpoints so that the example can be
>>>> easily run from the IDE, without requiring a local flink installation.
>>>>
>>>> The normal use for checkpoints is for failure recovery, while
>>>> savepoints are typically used for redeployments and rescaling -- and in
>>>> these cases the state processor API is not involved. You would use "flink
>>>> run -s ..." on the command line to manually resume from a checkpoint or
>>>> savepoint, and in the case of a job failure, the restart will happen
>>>> automatically.
>>>>
>>>> The flink operations playground [1] is a great way to gain more
>>>> understanding of these aspects of flink.
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/try-flink/flink-operations-playground.html
>>>>
>>>> Best regards,
>>>> David
>>>>
>>>> On Fri, Apr 30, 2021 at 1:56 PM Abdullah bin Omar <
>>>> abdullahbinomar29@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Please answer me some of my below question whether my understanding
>>>>> correct or not, and please answer the direct ask questions.
>>>>>
>>>>> *Question no 1 (about dependency):*
>>>>>
>>>>> *What is dependency (in pom.xml) for the org.apache.flink.training?*
>>>>>
>>>>> I am trying to *import* org.apache.flink.training.exercises.common.sources.TaxiFareGenerator;
>>>>> However, it can not resolve.
>>>>>
>>>>> [note that, I am using the group id: <groupId>org.apache.flink</
>>>>> groupId>
>>>>>
>>>>>
>>>>> *Question No 2 (which one is being load to an existing savepoint):*
>>>>>
>>>>>
>>>>> According to my understanding after reading [1], the name
>>>>> "ExistingSavepoint" looks like that it will restore all previous savepoint.
>>>>> However, according to [2], the input file is only a checkpointed file.
>>>>>
>>>>>
>>>>> *(i)* *is that mean that we can only load the last checkpointed file
>>>>> (in case of job failure) by using the ExistingSavepoint to restart the job
>>>>> where it fails?*
>>>>>
>>>>>
>>>>> *(ii)* *and there is no option to load all previous savepoint. is
>>>>> this correct?*
>>>>>
>>>>>
>>>>>
>>>>> *Question No 3 (about loading an existing savepoint):*
>>>>>
>>>>> ExecutionEnvironment bEnv = ExecutionEnvironment.
>>>>> *getExecutionEnvironment*();
>>>>>
>>>>> ExistingSavepoint sp = Savepoint.*load*(bEnv, "hdfs://path/", new
>>>>> MemoryStateBackend);
>>>>>
>>>>>
>>>>>
>>>>> This is the code for loading an existing savepoint. However, I
>>>>> configure a file location in flink conf to save the savepoint. So then,
>>>>> each time the job is running. I use a command in the terminal, ./bin/flink
>>>>> savepoint jobid
>>>>>
>>>>> and the savepointed file saved in the file location (that is set up in
>>>>> flink conf).
>>>>>
>>>>>
>>>>> In this case, to load the savepoint, file location will be the
>>>>> location that set up in the flink conf and FileSystemBackend will have to
>>>>> use instead of MemoryStateBackend. *is this correct?*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [1]
>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>>
>>>>> [2]
>>>>> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thank you
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Apr 23, 2021 at 10:10 AM David Anderson <da...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Abdullah,
>>>>>>
>>>>>> ReadRidesAndFaresSnapshot [1] is an example that shows how to use the
>>>>>> State Processor API to display the contents of a snapshot taken while
>>>>>> running RidesAndFaresSolution [2].
>>>>>>
>>>>>> Hopefully that will help you get started.
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
>>>>>> [2]
>>>>>> https://github.com/ververica/flink-training/blob/master/rides-and-fares/src/solution/java/org/apache/flink/training/solutions/ridesandfares/RidesAndFaresSolution.java
>>>>>>
>>>>>> Best regards,
>>>>>> David
>>>>>>
>>>>>> On Fri, Apr 23, 2021 at 3:32 PM Abdullah bin Omar <
>>>>>> abdullahbinomar29@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thank you for your reply.
>>>>>>>
>>>>>>> I want to read the previous snapshot (if needed) at the time of
>>>>>>> operation. In [1], there is a portion:
>>>>>>>
>>>>>>> DataSet<Integer> listState = savepoint.readListState<>(
>>>>>>> "my-uid",
>>>>>>> "list-state",
>>>>>>> Types.INT);
>>>>>>>
>>>>>>>
>>>>>>> here, will the function savepoint.readliststate<> () work to read
>>>>>>> the previous snapshot? If it is, then is the filename of a savepoint file
>>>>>>> similar to my-uid?
>>>>>>>
>>>>>>> [1]
>>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>>>>
>>>>>>> Thank you
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Apr 23, 2021 at 1:11 AM Matthias Pohl <
>>>>>>> matthias@ververica.com> wrote:
>>>>>>>
>>>>>>>> What is it you're trying to achieve in general? The JavaDoc of
>>>>>>>> MetadataV2V3SerializerBase provides a description on the format of the
>>>>>>>> file. Theoretically, you could come up with custom code using the Flink
>>>>>>>> sources to parse the content of the file. But maybe, there's another way to
>>>>>>>> accomplish what you're trying to do.
>>>>>>>>
>>>>>>>> Matthias
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>>>>>>
>>>>>>>> On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar <
>>>>>>>> abdullahbinomar29@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have a savepoint or checkpointed file from my task. However, the
>>>>>>>>> file is binary. I want to see what the file contains.
>>>>>>>>>
>>>>>>>>> How is it possible to see what information the file has (or how it
>>>>>>>>> is possible to make it human readable?)
>>>>>>>>>
>>>>>>>>> Thank you
>>>>>>>>>
>>>>>>>>> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <
>>>>>>>>> matthias@ververica.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Abdullah,
>>>>>>>>>> the metadata file contains handles to the operator states of the
>>>>>>>>>> checkpoint [1]. You might want to have a look into the State Processor API
>>>>>>>>>> [2].
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Matthias
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>>>>>>>> [2]
>>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
>>>>>>>>>> abdullahbinomar29@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> (1) what 's the snapshot metadata file (binary) contains ? is it
>>>>>>>>>>> possible to read the snapshot metadata file by using Flink Deserialization?
>>>>>>>>>>>
>>>>>>>>>>> (2) is there any function that can be used to see the previous
>>>>>>>>>>> states on time of operation?
>>>>>>>>>>>
>>>>>>>>>>> Thank you
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
Re: Question about snapshot file
Posted by David Anderson <da...@apache.org>.
>
> So, can't we extract all previous savepoint data by using
> ExistingSavepoint?
You can extract all of the data from any specific savepoint. Or nearly all
data, anyway. There is at least one corner case that isn't covered --
ListCheckpointed state -- which has been deprecated and isn't supported by
the savepoint API.
David
On Fri, Apr 30, 2021 at 5:42 PM Abdullah bin Omar <
abdullahbinomar29@gmail.com> wrote:
> Hi,
>
> So, can't we extract all previous savepoint data by using
> ExistingSavepoint?
>
>
> Thank you
>
>
>
>
>
>
> On Fri, Apr 30, 2021 at 10:25 AM David Anderson <da...@apache.org>
> wrote:
>
>> Abdullah,
>>
>> The example you are studying -- the one using the state processor API --
>> can be used with any retained checkpoint or savepoint created while running
>> the RidesAndFaresSolution job. But this is a very special use of
>> checkpoints and savepoints that shows how to extract data from them.
>>
>> Normally the state processor API is used with savepoints, and not with
>> checkpoints. This example uses checkpoints so that the example can be
>> easily run from the IDE, without requiring a local flink installation.
>>
>> The normal use for checkpoints is for failure recovery, while savepoints
>> are typically used for redeployments and rescaling -- and in these cases
>> the state processor API is not involved. You would use "flink run -s ..."
>> on the command line to manually resume from a checkpoint or savepoint, and
>> in the case of a job failure, the restart will happen automatically.
>>
>> The flink operations playground [1] is a great way to gain more
>> understanding of these aspects of flink.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/try-flink/flink-operations-playground.html
>>
>> Best regards,
>> David
>>
>> On Fri, Apr 30, 2021 at 1:56 PM Abdullah bin Omar <
>> abdullahbinomar29@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Please answer me some of my below question whether my understanding
>>> correct or not, and please answer the direct ask questions.
>>>
>>> *Question no 1 (about dependency):*
>>>
>>> *What is dependency (in pom.xml) for the org.apache.flink.training?*
>>>
>>> I am trying to *import* org.apache.flink.training.exercises.common.sources.TaxiFareGenerator;
>>> However, it can not resolve.
>>>
>>> [note that, I am using the group id: <groupId>org.apache.flink</groupId>
>>>
>>>
>>> *Question No 2 (which one is being load to an existing savepoint):*
>>>
>>>
>>> According to my understanding after reading [1], the name
>>> "ExistingSavepoint" looks like that it will restore all previous savepoint.
>>> However, according to [2], the input file is only a checkpointed file.
>>>
>>>
>>> *(i)* *is that mean that we can only load the last checkpointed file
>>> (in case of job failure) by using the ExistingSavepoint to restart the job
>>> where it fails?*
>>>
>>>
>>> *(ii)* *and there is no option to load all previous savepoint. is this
>>> correct?*
>>>
>>>
>>>
>>> *Question No 3 (about loading an existing savepoint):*
>>>
>>> ExecutionEnvironment bEnv = ExecutionEnvironment.
>>> *getExecutionEnvironment*();
>>>
>>> ExistingSavepoint sp = Savepoint.*load*(bEnv, "hdfs://path/", new
>>> MemoryStateBackend);
>>>
>>>
>>>
>>> This is the code for loading an existing savepoint. However, I configure
>>> a file location in flink conf to save the savepoint. So then, each time the
>>> job is running. I use a command in the terminal, ./bin/flink savepoint jobid
>>>
>>> and the savepointed file saved in the file location (that is set up in
>>> flink conf).
>>>
>>>
>>> In this case, to load the savepoint, file location will be the location
>>> that set up in the flink conf and FileSystemBackend will have to use
>>> instead of MemoryStateBackend. *is this correct?*
>>>
>>>
>>>
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>
>>> [2]
>>> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
>>>
>>>
>>>
>>>
>>> Thank you
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Apr 23, 2021 at 10:10 AM David Anderson <da...@apache.org>
>>> wrote:
>>>
>>>> Abdullah,
>>>>
>>>> ReadRidesAndFaresSnapshot [1] is an example that shows how to use the
>>>> State Processor API to display the contents of a snapshot taken while
>>>> running RidesAndFaresSolution [2].
>>>>
>>>> Hopefully that will help you get started.
>>>>
>>>> [1]
>>>> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
>>>> [2]
>>>> https://github.com/ververica/flink-training/blob/master/rides-and-fares/src/solution/java/org/apache/flink/training/solutions/ridesandfares/RidesAndFaresSolution.java
>>>>
>>>> Best regards,
>>>> David
>>>>
>>>> On Fri, Apr 23, 2021 at 3:32 PM Abdullah bin Omar <
>>>> abdullahbinomar29@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thank you for your reply.
>>>>>
>>>>> I want to read the previous snapshot (if needed) at the time of
>>>>> operation. In [1], there is a portion:
>>>>>
>>>>> DataSet<Integer> listState = savepoint.readListState<>(
>>>>> "my-uid",
>>>>> "list-state",
>>>>> Types.INT);
>>>>>
>>>>>
>>>>> here, will the function savepoint.readliststate<> () work to read the
>>>>> previous snapshot? If it is, then is the filename of a savepoint file
>>>>> similar to my-uid?
>>>>>
>>>>> [1]
>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>>
>>>>> Thank you
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Apr 23, 2021 at 1:11 AM Matthias Pohl <ma...@ververica.com>
>>>>> wrote:
>>>>>
>>>>>> What is it you're trying to achieve in general? The JavaDoc of
>>>>>> MetadataV2V3SerializerBase provides a description on the format of the
>>>>>> file. Theoretically, you could come up with custom code using the Flink
>>>>>> sources to parse the content of the file. But maybe, there's another way to
>>>>>> accomplish what you're trying to do.
>>>>>>
>>>>>> Matthias
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>>>>
>>>>>> On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar <
>>>>>> abdullahbinomar29@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have a savepoint or checkpointed file from my task. However, the
>>>>>>> file is binary. I want to see what the file contains.
>>>>>>>
>>>>>>> How is it possible to see what information the file has (or how it
>>>>>>> is possible to make it human readable?)
>>>>>>>
>>>>>>> Thank you
>>>>>>>
>>>>>>> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <
>>>>>>> matthias@ververica.com> wrote:
>>>>>>>
>>>>>>>> Hi Abdullah,
>>>>>>>> the metadata file contains handles to the operator states of the
>>>>>>>> checkpoint [1]. You might want to have a look into the State Processor API
>>>>>>>> [2].
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Matthias
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>>>>>> [2]
>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>>>>>
>>>>>>>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
>>>>>>>> abdullahbinomar29@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> (1) what 's the snapshot metadata file (binary) contains ? is it
>>>>>>>>> possible to read the snapshot metadata file by using Flink Deserialization?
>>>>>>>>>
>>>>>>>>> (2) is there any function that can be used to see the previous
>>>>>>>>> states on time of operation?
>>>>>>>>>
>>>>>>>>> Thank you
>>>>>>>>>
>>>>>>>>
>>>>>>
Re: Question about snapshot file
Posted by Abdullah bin Omar <ab...@gmail.com>.
Hi,
So, can't we extract all previous savepoint data by using
ExistingSavepoint?
Thank you
On Fri, Apr 30, 2021 at 10:25 AM David Anderson <da...@apache.org>
wrote:
> Abdullah,
>
> The example you are studying -- the one using the state processor API --
> can be used with any retained checkpoint or savepoint created while running
> the RidesAndFaresSolution job. But this is a very special use of
> checkpoints and savepoints that shows how to extract data from them.
>
> Normally the state processor API is used with savepoints, and not with
> checkpoints. This example uses checkpoints so that the example can be
> easily run from the IDE, without requiring a local flink installation.
>
> The normal use for checkpoints is for failure recovery, while savepoints
> are typically used for redeployments and rescaling -- and in these cases
> the state processor API is not involved. You would use "flink run -s ..."
> on the command line to manually resume from a checkpoint or savepoint, and
> in the case of a job failure, the restart will happen automatically.
>
> The flink operations playground [1] is a great way to gain more
> understanding of these aspects of flink.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/try-flink/flink-operations-playground.html
>
> Best regards,
> David
>
> On Fri, Apr 30, 2021 at 1:56 PM Abdullah bin Omar <
> abdullahbinomar29@gmail.com> wrote:
>
>> Hi,
>>
>> Please answer me some of my below question whether my understanding
>> correct or not, and please answer the direct ask questions.
>>
>> *Question no 1 (about dependency):*
>>
>> *What is dependency (in pom.xml) for the org.apache.flink.training?*
>>
>> I am trying to *import* org.apache.flink.training.exercises.common.sources.TaxiFareGenerator;
>> However, it can not resolve.
>>
>> [note that, I am using the group id: <groupId>org.apache.flink</groupId>
>>
>>
>> *Question No 2 (which one is being load to an existing savepoint):*
>>
>>
>> According to my understanding after reading [1], the name
>> "ExistingSavepoint" looks like that it will restore all previous savepoint.
>> However, according to [2], the input file is only a checkpointed file.
>>
>>
>> *(i)* *is that mean that we can only load the last checkpointed file (in
>> case of job failure) by using the ExistingSavepoint to restart the job
>> where it fails?*
>>
>>
>> *(ii)* *and there is no option to load all previous savepoint. is this
>> correct?*
>>
>>
>>
>> *Question No 3 (about loading an existing savepoint):*
>>
>> ExecutionEnvironment bEnv = ExecutionEnvironment.
>> *getExecutionEnvironment*();
>>
>> ExistingSavepoint sp = Savepoint.*load*(bEnv, "hdfs://path/", new
>> MemoryStateBackend);
>>
>>
>>
>> This is the code for loading an existing savepoint. However, I configure
>> a file location in flink conf to save the savepoint. So then, each time the
>> job is running. I use a command in the terminal, ./bin/flink savepoint jobid
>>
>> and the savepointed file saved in the file location (that is set up in
>> flink conf).
>>
>>
>> In this case, to load the savepoint, file location will be the location
>> that set up in the flink conf and FileSystemBackend will have to use
>> instead of MemoryStateBackend. *is this correct?*
>>
>>
>>
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>
>> [2]
>> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
>>
>>
>>
>>
>> Thank you
>>
>>
>>
>>
>>
>>
>> On Fri, Apr 23, 2021 at 10:10 AM David Anderson <da...@apache.org>
>> wrote:
>>
>>> Abdullah,
>>>
>>> ReadRidesAndFaresSnapshot [1] is an example that shows how to use the
>>> State Processor API to display the contents of a snapshot taken while
>>> running RidesAndFaresSolution [2].
>>>
>>> Hopefully that will help you get started.
>>>
>>> [1]
>>> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
>>> [2]
>>> https://github.com/ververica/flink-training/blob/master/rides-and-fares/src/solution/java/org/apache/flink/training/solutions/ridesandfares/RidesAndFaresSolution.java
>>>
>>> Best regards,
>>> David
>>>
>>> On Fri, Apr 23, 2021 at 3:32 PM Abdullah bin Omar <
>>> abdullahbinomar29@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thank you for your reply.
>>>>
>>>> I want to read the previous snapshot (if needed) at the time of
>>>> operation. In [1], there is a portion:
>>>>
>>>> DataSet<Integer> listState = savepoint.readListState<>(
>>>> "my-uid",
>>>> "list-state",
>>>> Types.INT);
>>>>
>>>>
>>>> here, will the function savepoint.readliststate<> () work to read the
>>>> previous snapshot? If it is, then is the filename of a savepoint file
>>>> similar to my-uid?
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>
>>>> Thank you
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Apr 23, 2021 at 1:11 AM Matthias Pohl <ma...@ververica.com>
>>>> wrote:
>>>>
>>>>> What is it you're trying to achieve in general? The JavaDoc of
>>>>> MetadataV2V3SerializerBase provides a description on the format of the
>>>>> file. Theoretically, you could come up with custom code using the Flink
>>>>> sources to parse the content of the file. But maybe, there's another way to
>>>>> accomplish what you're trying to do.
>>>>>
>>>>> Matthias
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>>>
>>>>> On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar <
>>>>> abdullahbinomar29@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a savepoint or checkpointed file from my task. However, the
>>>>>> file is binary. I want to see what the file contains.
>>>>>>
>>>>>> How is it possible to see what information the file has (or how it is
>>>>>> possible to make it human readable?)
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <
>>>>>> matthias@ververica.com> wrote:
>>>>>>
>>>>>>> Hi Abdullah,
>>>>>>> the metadata file contains handles to the operator states of the
>>>>>>> checkpoint [1]. You might want to have a look into the State Processor API
>>>>>>> [2].
>>>>>>>
>>>>>>> Best,
>>>>>>> Matthias
>>>>>>>
>>>>>>> [1]
>>>>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>>>>> [2]
>>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>>>>
>>>>>>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
>>>>>>> abdullahbinomar29@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> (1) what 's the snapshot metadata file (binary) contains ? is it
>>>>>>>> possible to read the snapshot metadata file by using Flink Deserialization?
>>>>>>>>
>>>>>>>> (2) is there any function that can be used to see the previous
>>>>>>>> states on time of operation?
>>>>>>>>
>>>>>>>> Thank you
>>>>>>>>
>>>>>>>
>>>>>
Re: Question about snapshot file
Posted by David Anderson <da...@apache.org>.
Abdullah,
The example you are studying -- the one using the state processor API --
can be used with any retained checkpoint or savepoint created while running
the RidesAndFaresSolution job. But this is a very special use of
checkpoints and savepoints that shows how to extract data from them.
Normally the state processor API is used with savepoints, and not with
checkpoints. This example uses checkpoints so that the example can be
easily run from the IDE, without requiring a local flink installation.
The normal use for checkpoints is for failure recovery, while savepoints
are typically used for redeployments and rescaling -- and in these cases
the state processor API is not involved. You would use "flink run -s ..."
on the command line to manually resume from a checkpoint or savepoint, and
in the case of a job failure, the restart will happen automatically.
The flink operations playground [1] is a great way to gain more
understanding of these aspects of flink.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/try-flink/flink-operations-playground.html
Best regards,
David
On Fri, Apr 30, 2021 at 1:56 PM Abdullah bin Omar <
abdullahbinomar29@gmail.com> wrote:
> Hi,
>
> Please answer me some of my below question whether my understanding
> correct or not, and please answer the direct ask questions.
>
> *Question no 1 (about dependency):*
>
> *What is dependency (in pom.xml) for the org.apache.flink.training?*
>
> I am trying to *import* org.apache.flink.training.exercises.common.sources.TaxiFareGenerator;
> However, it can not resolve.
>
> [note that, I am using the group id: <groupId>org.apache.flink</groupId>
>
>
> *Question No 2 (which one is being load to an existing savepoint):*
>
>
> According to my understanding after reading [1], the name
> "ExistingSavepoint" looks like that it will restore all previous savepoint.
> However, according to [2], the input file is only a checkpointed file.
>
>
> *(i)* *is that mean that we can only load the last checkpointed file (in
> case of job failure) by using the ExistingSavepoint to restart the job
> where it fails?*
>
>
> *(ii)* *and there is no option to load all previous savepoint. is this
> correct?*
>
>
>
> *Question No 3 (about loading an existing savepoint):*
>
> ExecutionEnvironment bEnv = ExecutionEnvironment.*getExecutionEnvironment*
> ();
>
> ExistingSavepoint sp = Savepoint.*load*(bEnv, "hdfs://path/", new
> MemoryStateBackend);
>
>
>
> This is the code for loading an existing savepoint. However, I configure a
> file location in flink conf to save the savepoint. So then, each time the
> job is running. I use a command in the terminal, ./bin/flink savepoint jobid
>
> and the savepointed file saved in the file location (that is set up in
> flink conf).
>
>
> In this case, to load the savepoint, file location will be the location
> that set up in the flink conf and FileSystemBackend will have to use
> instead of MemoryStateBackend. *is this correct?*
>
>
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>
> [2]
> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
>
>
>
>
> Thank you
>
>
>
>
>
>
> On Fri, Apr 23, 2021 at 10:10 AM David Anderson <da...@apache.org>
> wrote:
>
>> Abdullah,
>>
>> ReadRidesAndFaresSnapshot [1] is an example that shows how to use the
>> State Processor API to display the contents of a snapshot taken while
>> running RidesAndFaresSolution [2].
>>
>> Hopefully that will help you get started.
>>
>> [1]
>> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
>> [2]
>> https://github.com/ververica/flink-training/blob/master/rides-and-fares/src/solution/java/org/apache/flink/training/solutions/ridesandfares/RidesAndFaresSolution.java
>>
>> Best regards,
>> David
>>
>> On Fri, Apr 23, 2021 at 3:32 PM Abdullah bin Omar <
>> abdullahbinomar29@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Thank you for your reply.
>>>
>>> I want to read the previous snapshot (if needed) at the time of
>>> operation. In [1], there is a portion:
>>>
>>> DataSet<Integer> listState = savepoint.readListState<>(
>>> "my-uid",
>>> "list-state",
>>> Types.INT);
>>>
>>>
>>> here, will the function savepoint.readliststate<> () work to read the
>>> previous snapshot? If it is, then is the filename of a savepoint file
>>> similar to my-uid?
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>
>>> Thank you
>>>
>>>
>>>
>>>
>>> On Fri, Apr 23, 2021 at 1:11 AM Matthias Pohl <ma...@ververica.com>
>>> wrote:
>>>
>>>> What is it you're trying to achieve in general? The JavaDoc of
>>>> MetadataV2V3SerializerBase provides a description on the format of the
>>>> file. Theoretically, you could come up with custom code using the Flink
>>>> sources to parse the content of the file. But maybe, there's another way to
>>>> accomplish what you're trying to do.
>>>>
>>>> Matthias
>>>>
>>>> [1]
>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>>
>>>> On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar <
>>>> abdullahbinomar29@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a savepoint or checkpointed file from my task. However, the
>>>>> file is binary. I want to see what the file contains.
>>>>>
>>>>> How is it possible to see what information the file has (or how it is
>>>>> possible to make it human readable?)
>>>>>
>>>>> Thank you
>>>>>
>>>>> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <ma...@ververica.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Abdullah,
>>>>>> the metadata file contains handles to the operator states of the
>>>>>> checkpoint [1]. You might want to have a look into the State Processor API
>>>>>> [2].
>>>>>>
>>>>>> Best,
>>>>>> Matthias
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>>>> [2]
>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>>>
>>>>>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
>>>>>> abdullahbinomar29@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> (1) what 's the snapshot metadata file (binary) contains ? is it
>>>>>>> possible to read the snapshot metadata file by using Flink Deserialization?
>>>>>>>
>>>>>>> (2) is there any function that can be used to see the previous
>>>>>>> states on time of operation?
>>>>>>>
>>>>>>> Thank you
>>>>>>>
>>>>>>
>>>>
Re: Question about snapshot file
Posted by Abdullah bin Omar <ab...@gmail.com>.
Hi,
Please answer me some of my below question whether my understanding correct
or not, and please answer the direct ask questions.
*Question no 1 (about dependency):*
*What is dependency (in pom.xml) for the org.apache.flink.training?*
I am trying to *import*
org.apache.flink.training.exercises.common.sources.TaxiFareGenerator;
However, it can not resolve.
[note that, I am using the group id: <groupId>org.apache.flink</groupId>
*Question No 2 (which one is being load to an existing savepoint):*
According to my understanding after reading [1], the name
"ExistingSavepoint" looks like that it will restore all previous savepoint.
However, according to [2], the input file is only a checkpointed file.
*(i)* *is that mean that we can only load the last checkpointed file (in
case of job failure) by using the ExistingSavepoint to restart the job
where it fails?*
*(ii)* *and there is no option to load all previous savepoint. is this
correct?*
*Question No 3 (about loading an existing savepoint):*
ExecutionEnvironment bEnv = ExecutionEnvironment.*getExecutionEnvironment*
();
ExistingSavepoint sp = Savepoint.*load*(bEnv, "hdfs://path/", new
MemoryStateBackend);
This is the code for loading an existing savepoint. However, I configure a
file location in flink conf to save the savepoint. So then, each time the
job is running. I use a command in the terminal, ./bin/flink savepoint jobid
and the savepointed file saved in the file location (that is set up in
flink conf).
In this case, to load the savepoint, file location will be the location
that set up in the flink conf and FileSystemBackend will have to use
instead of MemoryStateBackend. *is this correct?*
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
[2]
https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
Thank you
On Fri, Apr 23, 2021 at 10:10 AM David Anderson <da...@apache.org>
wrote:
> Abdullah,
>
> ReadRidesAndFaresSnapshot [1] is an example that shows how to use the
> State Processor API to display the contents of a snapshot taken while
> running RidesAndFaresSolution [2].
>
> Hopefully that will help you get started.
>
> [1]
> https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
> [2]
> https://github.com/ververica/flink-training/blob/master/rides-and-fares/src/solution/java/org/apache/flink/training/solutions/ridesandfares/RidesAndFaresSolution.java
>
> Best regards,
> David
>
> On Fri, Apr 23, 2021 at 3:32 PM Abdullah bin Omar <
> abdullahbinomar29@gmail.com> wrote:
>
>> Hi,
>>
>> Thank you for your reply.
>>
>> I want to read the previous snapshot (if needed) at the time of
>> operation. In [1], there is a portion:
>>
>> DataSet<Integer> listState = savepoint.readListState<>(
>> "my-uid",
>> "list-state",
>> Types.INT);
>>
>>
>> here, will the function savepoint.readliststate<> () work to read the
>> previous snapshot? If it is, then is the filename of a savepoint file
>> similar to my-uid?
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>
>> Thank you
>>
>>
>>
>>
>> On Fri, Apr 23, 2021 at 1:11 AM Matthias Pohl <ma...@ververica.com>
>> wrote:
>>
>>> What is it you're trying to achieve in general? The JavaDoc of
>>> MetadataV2V3SerializerBase provides a description on the format of the
>>> file. Theoretically, you could come up with custom code using the Flink
>>> sources to parse the content of the file. But maybe, there's another way to
>>> accomplish what you're trying to do.
>>>
>>> Matthias
>>>
>>> [1]
>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>
>>> On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar <
>>> abdullahbinomar29@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a savepoint or checkpointed file from my task. However, the file
>>>> is binary. I want to see what the file contains.
>>>>
>>>> How is it possible to see what information the file has (or how it is
>>>> possible to make it human readable?)
>>>>
>>>> Thank you
>>>>
>>>> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <ma...@ververica.com>
>>>> wrote:
>>>>
>>>>> Hi Abdullah,
>>>>> the metadata file contains handles to the operator states of the
>>>>> checkpoint [1]. You might want to have a look into the State Processor API
>>>>> [2].
>>>>>
>>>>> Best,
>>>>> Matthias
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>>> [2]
>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>>
>>>>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
>>>>> abdullahbinomar29@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> (1) what 's the snapshot metadata file (binary) contains ? is it
>>>>>> possible to read the snapshot metadata file by using Flink Deserialization?
>>>>>>
>>>>>> (2) is there any function that can be used to see the previous
>>>>>> states on time of operation?
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>
>>>
Re: Question about snapshot file
Posted by David Anderson <da...@apache.org>.
Abdullah,
ReadRidesAndFaresSnapshot [1] is an example that shows how to use the State
Processor API to display the contents of a snapshot taken while running
RidesAndFaresSolution [2].
Hopefully that will help you get started.
[1]
https://github.com/ververica/flink-training/blob/master/state-processor/src/main/java/com/ververica/flink/training/exercises/ReadRidesAndFaresSnapshot.java
[2]
https://github.com/ververica/flink-training/blob/master/rides-and-fares/src/solution/java/org/apache/flink/training/solutions/ridesandfares/RidesAndFaresSolution.java
Best regards,
David
On Fri, Apr 23, 2021 at 3:32 PM Abdullah bin Omar <
abdullahbinomar29@gmail.com> wrote:
> Hi,
>
> Thank you for your reply.
>
> I want to read the previous snapshot (if needed) at the time of operation.
> In [1], there is a portion:
>
> DataSet<Integer> listState = savepoint.readListState<>(
> "my-uid",
> "list-state",
> Types.INT);
>
>
> here, will the function savepoint.readliststate<> () work to read the
> previous snapshot? If it is, then is the filename of a savepoint file
> similar to my-uid?
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>
> Thank you
>
>
>
>
> On Fri, Apr 23, 2021 at 1:11 AM Matthias Pohl <ma...@ververica.com>
> wrote:
>
>> What is it you're trying to achieve in general? The JavaDoc of
>> MetadataV2V3SerializerBase provides a description on the format of the
>> file. Theoretically, you could come up with custom code using the Flink
>> sources to parse the content of the file. But maybe, there's another way to
>> accomplish what you're trying to do.
>>
>> Matthias
>>
>> [1]
>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>
>> On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar <
>> abdullahbinomar29@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a savepoint or checkpointed file from my task. However, the file
>>> is binary. I want to see what the file contains.
>>>
>>> How is it possible to see what information the file has (or how it is
>>> possible to make it human readable?)
>>>
>>> Thank you
>>>
>>> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <ma...@ververica.com>
>>> wrote:
>>>
>>>> Hi Abdullah,
>>>> the metadata file contains handles to the operator states of the
>>>> checkpoint [1]. You might want to have a look into the State Processor API
>>>> [2].
>>>>
>>>> Best,
>>>> Matthias
>>>>
>>>> [1]
>>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>>> [2]
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>>
>>>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
>>>> abdullahbinomar29@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> (1) what 's the snapshot metadata file (binary) contains ? is it
>>>>> possible to read the snapshot metadata file by using Flink Deserialization?
>>>>>
>>>>> (2) is there any function that can be used to see the previous
>>>>> states on time of operation?
>>>>>
>>>>> Thank you
>>>>>
>>>>
>>
Re: Question about snapshot file
Posted by Abdullah bin Omar <ab...@gmail.com>.
Hi,
Thank you for your reply.
I want to read the previous snapshot (if needed) at the time of operation.
In [1], there is a portion:
DataSet<Integer> listState = savepoint.readListState<>(
"my-uid",
"list-state",
Types.INT);
here, will the function savepoint.readliststate<> () work to read the
previous snapshot? If it is, then is the filename of a savepoint file
similar to my-uid?
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
Thank you
On Fri, Apr 23, 2021 at 1:11 AM Matthias Pohl <ma...@ververica.com>
wrote:
> What is it you're trying to achieve in general? The JavaDoc of
> MetadataV2V3SerializerBase provides a description on the format of the
> file. Theoretically, you could come up with custom code using the Flink
> sources to parse the content of the file. But maybe, there's another way to
> accomplish what you're trying to do.
>
> Matthias
>
> [1]
> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>
> On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar <
> abdullahbinomar29@gmail.com> wrote:
>
>> Hi,
>>
>> I have a savepoint or checkpointed file from my task. However, the file
>> is binary. I want to see what the file contains.
>>
>> How is it possible to see what information the file has (or how it is
>> possible to make it human readable?)
>>
>> Thank you
>>
>> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <ma...@ververica.com>
>> wrote:
>>
>>> Hi Abdullah,
>>> the metadata file contains handles to the operator states of the
>>> checkpoint [1]. You might want to have a look into the State Processor API
>>> [2].
>>>
>>> Best,
>>> Matthias
>>>
>>> [1]
>>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>>> [2]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>>
>>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
>>> abdullahbinomar29@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> (1) what 's the snapshot metadata file (binary) contains ? is it
>>>> possible to read the snapshot metadata file by using Flink Deserialization?
>>>>
>>>> (2) is there any function that can be used to see the previous
>>>> states on time of operation?
>>>>
>>>> Thank you
>>>>
>>>
>
Re: Question about snapshot file
Posted by Matthias Pohl <ma...@ververica.com>.
What is it you're trying to achieve in general? The JavaDoc of
MetadataV2V3SerializerBase provides a description on the format of the
file. Theoretically, you could come up with custom code using the Flink
sources to parse the content of the file. But maybe, there's another way to
accomplish what you're trying to do.
Matthias
[1]
https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
On Thu, Apr 22, 2021 at 7:53 PM Abdullah bin Omar <
abdullahbinomar29@gmail.com> wrote:
> Hi,
>
> I have a savepoint or checkpointed file from my task. However, the file is
> binary. I want to see what the file contains.
>
> How is it possible to see what information the file has (or how it is
> possible to make it human readable?)
>
> Thank you
>
> On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <ma...@ververica.com>
> wrote:
>
>> Hi Abdullah,
>> the metadata file contains handles to the operator states of the
>> checkpoint [1]. You might want to have a look into the State Processor API
>> [2].
>>
>> Best,
>> Matthias
>>
>> [1]
>> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
>> [2]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>
>> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
>> abdullahbinomar29@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> (1) what 's the snapshot metadata file (binary) contains ? is it
>>> possible to read the snapshot metadata file by using Flink Deserialization?
>>>
>>> (2) is there any function that can be used to see the previous states on
>>> time of operation?
>>>
>>> Thank you
>>>
>>
Re: Question about snapshot file
Posted by Abdullah bin Omar <ab...@gmail.com>.
Hi,
I have a savepoint or checkpointed file from my task. However, the file is
binary. I want to see what the file contains.
How is it possible to see what information the file has (or how it is
possible to make it human readable?)
Thank you
On Thu, Apr 22, 2021 at 10:19 AM Matthias Pohl <ma...@ververica.com>
wrote:
> Hi Abdullah,
> the metadata file contains handles to the operator states of the
> checkpoint [1]. You might want to have a look into the State Processor API
> [2].
>
> Best,
> Matthias
>
> [1]
> https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
> [2]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>
> On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
> abdullahbinomar29@gmail.com> wrote:
>
>> Hi,
>>
>> (1) what 's the snapshot metadata file (binary) contains ? is it possible
>> to read the snapshot metadata file by using Flink Deserialization?
>>
>> (2) is there any function that can be used to see the previous states on
>> time of operation?
>>
>> Thank you
>>
>
Re: Question about snapshot file
Posted by Matthias Pohl <ma...@ververica.com>.
Hi Abdullah,
the metadata file contains handles to the operator states of the checkpoint
[1]. You might want to have a look into the State Processor API [2].
Best,
Matthias
[1]
https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
On Thu, Apr 22, 2021 at 4:57 PM Abdullah bin Omar <
abdullahbinomar29@gmail.com> wrote:
> Hi,
>
> (1) what 's the snapshot metadata file (binary) contains ? is it possible
> to read the snapshot metadata file by using Flink Deserialization?
>
> (2) is there any function that can be used to see the previous states on
> time of operation?
>
> Thank you
>