You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Hao Sun <ha...@zendesk.com> on 2017/12/01 23:28:08 UTC
Trace jar file name from jobId, is that possible?
Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app.
I want to use savepoints to resume a job after each deployment.
As you know I need jar file name and path to savepoints to resume a task.
Currently `flink list` command only gives me job ids, not jar file names.
And REST API does not have that information as well. If I have multiple jar
files how can I map the savepoints back to jars, so I can resume my task?
I thought about save the jar to jid map somewhere, but Flink can reschedule
a task on failures, so the map will be stale.
Any thoughts is appreciated. Many thanks.
Re: Trace jar file name from jobId, is that possible?
Posted by Hao Sun <ha...@zendesk.com>.
Let me check details, on top of my mind I remember the job id changes, I
might be wrong.
On Thu, Dec 7, 2017, 08:48 Fabian Hueske <fh...@gmail.com> wrote:
> AFAIK, a job keeps its ID in case of a recovery.
> Did you observe something else?
>
> 2017-12-07 17:32 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>
>> I mean restarted during failure recovery
>>
>> On Thu, Dec 7, 2017 at 8:29 AM Fabian Hueske <fh...@gmail.com> wrote:
>>
>>> What do you mean by rescheduled?
>>> Started from a savepoint or restarted during failure recovery?
>>>
>>>
>>> 2017-12-07 16:59 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>>>
>>>> Anything I can do for the job reschedule case? Thanks.
>>>> Or is there a way to add job lifecycle hooks to trace it?
>>>>
>>>> On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <ha...@zendesk.com> wrote:
>>>>
>>>>> Thanks Fabian, there is one case can not be covered by the REST API.
>>>>> When a job rescheduled to run, but jobid will change, and I wont be able to
>>>>> backtrace the jar name. Why not keep the jar name stored somewhere and
>>>>> expose it through the api as well?
>>>>>
>>>>> On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <fh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> you can submit jar files and start jobs via the REST interface [1].
>>>>>> When starting a job, you get the jobId. You can link jar files and
>>>>>> savepoints via the jobId.
>>>>>>
>>>>>> Best, Fabian
>>>>>>
>>>>>> [1]
>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs
>>>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs>
>>>>>>
>>>>>> 2017-12-02 0:28 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>>>>>>
>>>>>>> Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for
>>>>>>> my app.
>>>>>>>
>>>>>>> I want to use savepoints to resume a job after each deployment.
>>>>>>> As you know I need jar file name and path to savepoints to resume a
>>>>>>> task.
>>>>>>>
>>>>>>> Currently `flink list` command only gives me job ids, not jar file
>>>>>>> names.
>>>>>>> And REST API does not have that information as well. If I have
>>>>>>> multiple jar files how can I map the savepoints back to jars, so I can
>>>>>>> resume my task?
>>>>>>>
>>>>>>> I thought about save the jar to jid map somewhere, but Flink can
>>>>>>> reschedule a task on failures, so the map will be stale.
>>>>>>>
>>>>>>> Any thoughts is appreciated. Many thanks.
>>>>>>>
>>>>>>>
>>>>>>
>>>
>
Re: Trace jar file name from jobId, is that possible?
Posted by Fabian Hueske <fh...@gmail.com>.
AFAIK, a job keeps its ID in case of a recovery.
Did you observe something else?
2017-12-07 17:32 GMT+01:00 Hao Sun <ha...@zendesk.com>:
> I mean restarted during failure recovery
>
> On Thu, Dec 7, 2017 at 8:29 AM Fabian Hueske <fh...@gmail.com> wrote:
>
>> What do you mean by rescheduled?
>> Started from a savepoint or restarted during failure recovery?
>>
>>
>> 2017-12-07 16:59 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>>
>>> Anything I can do for the job reschedule case? Thanks.
>>> Or is there a way to add job lifecycle hooks to trace it?
>>>
>>> On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <ha...@zendesk.com> wrote:
>>>
>>>> Thanks Fabian, there is one case can not be covered by the REST API.
>>>> When a job rescheduled to run, but jobid will change, and I wont be able to
>>>> backtrace the jar name. Why not keep the jar name stored somewhere and
>>>> expose it through the api as well?
>>>>
>>>> On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <fh...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> you can submit jar files and start jobs via the REST interface [1].
>>>>> When starting a job, you get the jobId. You can link jar files and
>>>>> savepoints via the jobId.
>>>>>
>>>>> Best, Fabian
>>>>>
>>>>> [1] https://ci.apache.org/projects/flink/flink-docs-
>>>>> release-1.3/monitoring/rest_api.html#submitting-programs
>>>>>
>>>>> 2017-12-02 0:28 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>>>>>
>>>>>> Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for
>>>>>> my app.
>>>>>>
>>>>>> I want to use savepoints to resume a job after each deployment.
>>>>>> As you know I need jar file name and path to savepoints to resume a
>>>>>> task.
>>>>>>
>>>>>> Currently `flink list` command only gives me job ids, not jar file
>>>>>> names.
>>>>>> And REST API does not have that information as well. If I have
>>>>>> multiple jar files how can I map the savepoints back to jars, so I can
>>>>>> resume my task?
>>>>>>
>>>>>> I thought about save the jar to jid map somewhere, but Flink can
>>>>>> reschedule a task on failures, so the map will be stale.
>>>>>>
>>>>>> Any thoughts is appreciated. Many thanks.
>>>>>>
>>>>>>
>>>>>
>>
Re: Trace jar file name from jobId, is that possible?
Posted by Hao Sun <ha...@zendesk.com>.
I mean restarted during failure recovery
On Thu, Dec 7, 2017 at 8:29 AM Fabian Hueske <fh...@gmail.com> wrote:
> What do you mean by rescheduled?
> Started from a savepoint or restarted during failure recovery?
>
>
> 2017-12-07 16:59 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>
>> Anything I can do for the job reschedule case? Thanks.
>> Or is there a way to add job lifecycle hooks to trace it?
>>
>> On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <ha...@zendesk.com> wrote:
>>
>>> Thanks Fabian, there is one case can not be covered by the REST API.
>>> When a job rescheduled to run, but jobid will change, and I wont be able to
>>> backtrace the jar name. Why not keep the jar name stored somewhere and
>>> expose it through the api as well?
>>>
>>> On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <fh...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> you can submit jar files and start jobs via the REST interface [1].
>>>> When starting a job, you get the jobId. You can link jar files and
>>>> savepoints via the jobId.
>>>>
>>>> Best, Fabian
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs
>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs>
>>>>
>>>> 2017-12-02 0:28 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>>>>
>>>>> Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for
>>>>> my app.
>>>>>
>>>>> I want to use savepoints to resume a job after each deployment.
>>>>> As you know I need jar file name and path to savepoints to resume a
>>>>> task.
>>>>>
>>>>> Currently `flink list` command only gives me job ids, not jar file
>>>>> names.
>>>>> And REST API does not have that information as well. If I have
>>>>> multiple jar files how can I map the savepoints back to jars, so I can
>>>>> resume my task?
>>>>>
>>>>> I thought about save the jar to jid map somewhere, but Flink can
>>>>> reschedule a task on failures, so the map will be stale.
>>>>>
>>>>> Any thoughts is appreciated. Many thanks.
>>>>>
>>>>>
>>>>
>
Re: Trace jar file name from jobId, is that possible?
Posted by Fabian Hueske <fh...@gmail.com>.
What do you mean by rescheduled?
Started from a savepoint or restarted during failure recovery?
2017-12-07 16:59 GMT+01:00 Hao Sun <ha...@zendesk.com>:
> Anything I can do for the job reschedule case? Thanks.
> Or is there a way to add job lifecycle hooks to trace it?
>
> On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <ha...@zendesk.com> wrote:
>
>> Thanks Fabian, there is one case can not be covered by the REST API. When
>> a job rescheduled to run, but jobid will change, and I wont be able to
>> backtrace the jar name. Why not keep the jar name stored somewhere and
>> expose it through the api as well?
>>
>> On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <fh...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> you can submit jar files and start jobs via the REST interface [1].
>>> When starting a job, you get the jobId. You can link jar files and
>>> savepoints via the jobId.
>>>
>>> Best, Fabian
>>>
>>> [1] https://ci.apache.org/projects/flink/flink-docs-
>>> release-1.3/monitoring/rest_api.html#submitting-programs
>>>
>>> 2017-12-02 0:28 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>>>
>>>> Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my
>>>> app.
>>>>
>>>> I want to use savepoints to resume a job after each deployment.
>>>> As you know I need jar file name and path to savepoints to resume a
>>>> task.
>>>>
>>>> Currently `flink list` command only gives me job ids, not jar file
>>>> names.
>>>> And REST API does not have that information as well. If I have multiple
>>>> jar files how can I map the savepoints back to jars, so I can resume my
>>>> task?
>>>>
>>>> I thought about save the jar to jid map somewhere, but Flink can
>>>> reschedule a task on failures, so the map will be stale.
>>>>
>>>> Any thoughts is appreciated. Many thanks.
>>>>
>>>>
>>>
Re: Trace jar file name from jobId, is that possible?
Posted by Hao Sun <ha...@zendesk.com>.
Anything I can do for the job reschedule case? Thanks.
Or is there a way to add job lifecycle hooks to trace it?
On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <ha...@zendesk.com> wrote:
> Thanks Fabian, there is one case can not be covered by the REST API. When
> a job rescheduled to run, but jobid will change, and I wont be able to
> backtrace the jar name. Why not keep the jar name stored somewhere and
> expose it through the api as well?
>
> On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi,
>>
>> you can submit jar files and start jobs via the REST interface [1].
>> When starting a job, you get the jobId. You can link jar files and
>> savepoints via the jobId.
>>
>> Best, Fabian
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs>
>>
>> 2017-12-02 0:28 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>>
>>> Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my
>>> app.
>>>
>>> I want to use savepoints to resume a job after each deployment.
>>> As you know I need jar file name and path to savepoints to resume a task.
>>>
>>> Currently `flink list` command only gives me job ids, not jar file names.
>>> And REST API does not have that information as well. If I have multiple
>>> jar files how can I map the savepoints back to jars, so I can resume my
>>> task?
>>>
>>> I thought about save the jar to jid map somewhere, but Flink can
>>> reschedule a task on failures, so the map will be stale.
>>>
>>> Any thoughts is appreciated. Many thanks.
>>>
>>>
>>
Re: Trace jar file name from jobId, is that possible?
Posted by Hao Sun <ha...@zendesk.com>.
Thanks Fabian, there is one case can not be covered by the REST API. When a
job rescheduled to run, but jobid will change, and I wont be able to
backtrace the jar name. Why not keep the jar name stored somewhere and
expose it through the api as well?
On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <fh...@gmail.com> wrote:
> Hi,
>
> you can submit jar files and start jobs via the REST interface [1].
> When starting a job, you get the jobId. You can link jar files and
> savepoints via the jobId.
>
> Best, Fabian
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs
> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs>
>
> 2017-12-02 0:28 GMT+01:00 Hao Sun <ha...@zendesk.com>:
>
>> Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my
>> app.
>>
>> I want to use savepoints to resume a job after each deployment.
>> As you know I need jar file name and path to savepoints to resume a task.
>>
>> Currently `flink list` command only gives me job ids, not jar file names.
>> And REST API does not have that information as well. If I have multiple
>> jar files how can I map the savepoints back to jars, so I can resume my
>> task?
>>
>> I thought about save the jar to jid map somewhere, but Flink can
>> reschedule a task on failures, so the map will be stale.
>>
>> Any thoughts is appreciated. Many thanks.
>>
>>
>
Re: Trace jar file name from jobId, is that possible?
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
you can submit jar files and start jobs via the REST interface [1].
When starting a job, you get the jobId. You can link jar files and
savepoints via the jobId.
Best, Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs
2017-12-02 0:28 GMT+01:00 Hao Sun <ha...@zendesk.com>:
> Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my
> app.
>
> I want to use savepoints to resume a job after each deployment.
> As you know I need jar file name and path to savepoints to resume a task.
>
> Currently `flink list` command only gives me job ids, not jar file names.
> And REST API does not have that information as well. If I have multiple
> jar files how can I map the savepoints back to jars, so I can resume my
> task?
>
> I thought about save the jar to jid map somewhere, but Flink can
> reschedule a task on failures, so the map will be stale.
>
> Any thoughts is appreciated. Many thanks.
>
>