You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Filip Łęczycki <fi...@gmail.com> on 2016/01/01 23:29:23 UTC
Monitoring single-run job statistics
Hi all,
I am running filnk aps on YARN cluster and I am trying to get some
benchmarks. When I start a long-running flink cluster on my YARN cluster I
have an access to web UI and rest API that provide me statistics of the
deployed jobs (as desribed here:
https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html).
I was wondering is this possible to get such information about a single run
job trigerred with 'flink run -m yarn-cluster ...'? After the job is
finished there is no flink client running so I cannot use rest api to get
stats.
Thanks for any help:)
Best regards/Pozdrawiam,
Filip Łęczycki
Re: Monitoring single-run job statistics
Posted by Stephan Ewen <se...@apache.org>.
Good luck with the Thesis!
On Wed, Jan 6, 2016 at 6:18 PM, Filip Łęczycki <fi...@gmail.com>
wrote:
> Hi Stephan,
>
> Thank you for you answer. I would love to contribute but currently I have
> no capacity as I am buried with my thesis.
>
> I will reach out after graduating :)
>
> Bestr regards.
> Filip
>
> Pozdrawiam,
> Filip Łęczycki
>
> 2016-01-05 10:35 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
>> Hi Filip!
>>
>> There are thoughts and efforts to extend Flink to push the result
>> statistics of Flink jobs to the YARN timeline server. That way, you can
>> explore jobs that are completed.
>>
>> Since the whole web dashboard in Flink has a pure REST design, this is a
>> quite straightforward fix.
>>
>> From the capacities I see in the community, I can not promise that to be
>> fixed immediately. Let me know, though, if you are interested in
>> contributing an addition there, and I can walk you through the steps that
>> would be needed.
>>
>> Greetings,
>> Stephan
>>
>>
>> On Mon, Jan 4, 2016 at 9:17 PM, Filip Łęczycki <fi...@gmail.com>
>> wrote:
>>
>>> Hi Till,
>>>
>>> Thank you for you answer however I am sorry to hear that. I was
>>> reluctant to execute jobs with long running Flink cluster due to the fact
>>> that multiple jobs would cloud yarn statistics regarding cpu and memory
>>> time as well as Flink's garbage collector statistics in log, as they would
>>> be stored for the whole Flink cluster, instead of a single job.
>>>
>>> Do you know whether is there a way to extract mentioned stats (cpu time,
>>> mem time, gc time) for a single job ran on long running Flink cluster?
>>>
>>> I will be very grateful for an answer:)
>>>
>>> Best regards,
>>> Filip
>>>
>>> Pozdrawiam,
>>> Filip Łęczycki
>>>
>>> 2016-01-04 10:05 GMT+01:00 Till Rohrmann <ti...@gmail.com>:
>>>
>>>> Hi Filip,
>>>>
>>>> at the moment it is not possible to retrieve the job statistics after
>>>> the job has finished with flink run -m yarn-cluster. The reason is
>>>> that the YARN cluster is only alive as long as the job is executed. Thus, I
>>>> would recommend you to execute your jobs with a long running Flink cluster
>>>> on YARN.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>>
>>>> On Fri, Jan 1, 2016 at 11:29 PM, Filip Łęczycki <
>>>> filipleczycki@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I am running filnk aps on YARN cluster and I am trying to get some
>>>>> benchmarks. When I start a long-running flink cluster on my YARN cluster I
>>>>> have an access to web UI and rest API that provide me statistics of the
>>>>> deployed jobs (as desribed here:
>>>>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html).
>>>>> I was wondering is this possible to get such information about a single run
>>>>> job trigerred with 'flink run -m yarn-cluster ...'? After the job is
>>>>> finished there is no flink client running so I cannot use rest api to get
>>>>> stats.
>>>>>
>>>>> Thanks for any help:)
>>>>>
>>>>>
>>>>> Best regards/Pozdrawiam,
>>>>> Filip Łęczycki
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Monitoring single-run job statistics
Posted by Filip Łęczycki <fi...@gmail.com>.
Hi Stephan,
Thank you for you answer. I would love to contribute but currently I have
no capacity as I am buried with my thesis.
I will reach out after graduating :)
Bestr regards.
Filip
Pozdrawiam,
Filip Łęczycki
2016-01-05 10:35 GMT+01:00 Stephan Ewen <se...@apache.org>:
> Hi Filip!
>
> There are thoughts and efforts to extend Flink to push the result
> statistics of Flink jobs to the YARN timeline server. That way, you can
> explore jobs that are completed.
>
> Since the whole web dashboard in Flink has a pure REST design, this is a
> quite straightforward fix.
>
> From the capacities I see in the community, I can not promise that to be
> fixed immediately. Let me know, though, if you are interested in
> contributing an addition there, and I can walk you through the steps that
> would be needed.
>
> Greetings,
> Stephan
>
>
> On Mon, Jan 4, 2016 at 9:17 PM, Filip Łęczycki <fi...@gmail.com>
> wrote:
>
>> Hi Till,
>>
>> Thank you for you answer however I am sorry to hear that. I was reluctant
>> to execute jobs with long running Flink cluster due to the fact that
>> multiple jobs would cloud yarn statistics regarding cpu and memory time as
>> well as Flink's garbage collector statistics in log, as they would be
>> stored for the whole Flink cluster, instead of a single job.
>>
>> Do you know whether is there a way to extract mentioned stats (cpu time,
>> mem time, gc time) for a single job ran on long running Flink cluster?
>>
>> I will be very grateful for an answer:)
>>
>> Best regards,
>> Filip
>>
>> Pozdrawiam,
>> Filip Łęczycki
>>
>> 2016-01-04 10:05 GMT+01:00 Till Rohrmann <ti...@gmail.com>:
>>
>>> Hi Filip,
>>>
>>> at the moment it is not possible to retrieve the job statistics after
>>> the job has finished with flink run -m yarn-cluster. The reason is that
>>> the YARN cluster is only alive as long as the job is executed. Thus, I
>>> would recommend you to execute your jobs with a long running Flink cluster
>>> on YARN.
>>>
>>> Cheers,
>>> Till
>>>
>>>
>>> On Fri, Jan 1, 2016 at 11:29 PM, Filip Łęczycki <filipleczycki@gmail.com
>>> > wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am running filnk aps on YARN cluster and I am trying to get some
>>>> benchmarks. When I start a long-running flink cluster on my YARN cluster I
>>>> have an access to web UI and rest API that provide me statistics of the
>>>> deployed jobs (as desribed here:
>>>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html).
>>>> I was wondering is this possible to get such information about a single run
>>>> job trigerred with 'flink run -m yarn-cluster ...'? After the job is
>>>> finished there is no flink client running so I cannot use rest api to get
>>>> stats.
>>>>
>>>> Thanks for any help:)
>>>>
>>>>
>>>> Best regards/Pozdrawiam,
>>>> Filip Łęczycki
>>>>
>>>
>>>
>>
>
Re: Monitoring single-run job statistics
Posted by Stephan Ewen <se...@apache.org>.
Hi Filip!
There are thoughts and efforts to extend Flink to push the result
statistics of Flink jobs to the YARN timeline server. That way, you can
explore jobs that are completed.
Since the whole web dashboard in Flink has a pure REST design, this is a
quite straightforward fix.
>From the capacities I see in the community, I can not promise that to be
fixed immediately. Let me know, though, if you are interested in
contributing an addition there, and I can walk you through the steps that
would be needed.
Greetings,
Stephan
On Mon, Jan 4, 2016 at 9:17 PM, Filip Łęczycki <fi...@gmail.com>
wrote:
> Hi Till,
>
> Thank you for you answer however I am sorry to hear that. I was reluctant
> to execute jobs with long running Flink cluster due to the fact that
> multiple jobs would cloud yarn statistics regarding cpu and memory time as
> well as Flink's garbage collector statistics in log, as they would be
> stored for the whole Flink cluster, instead of a single job.
>
> Do you know whether is there a way to extract mentioned stats (cpu time,
> mem time, gc time) for a single job ran on long running Flink cluster?
>
> I will be very grateful for an answer:)
>
> Best regards,
> Filip
>
> Pozdrawiam,
> Filip Łęczycki
>
> 2016-01-04 10:05 GMT+01:00 Till Rohrmann <ti...@gmail.com>:
>
>> Hi Filip,
>>
>> at the moment it is not possible to retrieve the job statistics after the
>> job has finished with flink run -m yarn-cluster. The reason is that the
>> YARN cluster is only alive as long as the job is executed. Thus, I would
>> recommend you to execute your jobs with a long running Flink cluster on
>> YARN.
>>
>> Cheers,
>> Till
>>
>>
>> On Fri, Jan 1, 2016 at 11:29 PM, Filip Łęczycki <fi...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I am running filnk aps on YARN cluster and I am trying to get some
>>> benchmarks. When I start a long-running flink cluster on my YARN cluster I
>>> have an access to web UI and rest API that provide me statistics of the
>>> deployed jobs (as desribed here:
>>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html).
>>> I was wondering is this possible to get such information about a single run
>>> job trigerred with 'flink run -m yarn-cluster ...'? After the job is
>>> finished there is no flink client running so I cannot use rest api to get
>>> stats.
>>>
>>> Thanks for any help:)
>>>
>>>
>>> Best regards/Pozdrawiam,
>>> Filip Łęczycki
>>>
>>
>>
>
Re: Monitoring single-run job statistics
Posted by Filip Łęczycki <fi...@gmail.com>.
Hi Till,
Thank you for you answer however I am sorry to hear that. I was reluctant
to execute jobs with long running Flink cluster due to the fact that
multiple jobs would cloud yarn statistics regarding cpu and memory time as
well as Flink's garbage collector statistics in log, as they would be
stored for the whole Flink cluster, instead of a single job.
Do you know whether is there a way to extract mentioned stats (cpu time,
mem time, gc time) for a single job ran on long running Flink cluster?
I will be very grateful for an answer:)
Best regards,
Filip
Pozdrawiam,
Filip Łęczycki
2016-01-04 10:05 GMT+01:00 Till Rohrmann <ti...@gmail.com>:
> Hi Filip,
>
> at the moment it is not possible to retrieve the job statistics after the
> job has finished with flink run -m yarn-cluster. The reason is that the
> YARN cluster is only alive as long as the job is executed. Thus, I would
> recommend you to execute your jobs with a long running Flink cluster on
> YARN.
>
> Cheers,
> Till
>
>
> On Fri, Jan 1, 2016 at 11:29 PM, Filip Łęczycki <fi...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I am running filnk aps on YARN cluster and I am trying to get some
>> benchmarks. When I start a long-running flink cluster on my YARN cluster I
>> have an access to web UI and rest API that provide me statistics of the
>> deployed jobs (as desribed here:
>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html).
>> I was wondering is this possible to get such information about a single run
>> job trigerred with 'flink run -m yarn-cluster ...'? After the job is
>> finished there is no flink client running so I cannot use rest api to get
>> stats.
>>
>> Thanks for any help:)
>>
>>
>> Best regards/Pozdrawiam,
>> Filip Łęczycki
>>
>
>
Re: Monitoring single-run job statistics
Posted by Till Rohrmann <ti...@gmail.com>.
Hi Filip,
at the moment it is not possible to retrieve the job statistics after the
job has finished with flink run -m yarn-cluster. The reason is that the
YARN cluster is only alive as long as the job is executed. Thus, I would
recommend you to execute your jobs with a long running Flink cluster on
YARN.
Cheers,
Till
On Fri, Jan 1, 2016 at 11:29 PM, Filip Łęczycki <fi...@gmail.com>
wrote:
> Hi all,
>
> I am running filnk aps on YARN cluster and I am trying to get some
> benchmarks. When I start a long-running flink cluster on my YARN cluster I
> have an access to web UI and rest API that provide me statistics of the
> deployed jobs (as desribed here:
> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html).
> I was wondering is this possible to get such information about a single run
> job trigerred with 'flink run -m yarn-cluster ...'? After the job is
> finished there is no flink client running so I cannot use rest api to get
> stats.
>
> Thanks for any help:)
>
>
> Best regards/Pozdrawiam,
> Filip Łęczycki
>