You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Philip Lee <ph...@gmail.com> on 2016/01/25 16:15:27 UTC
Hello, a question about Dashborad in Flink
Hello,
According to
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
I cannot retrieve the job history from Dashboard after turnning off JM.
But as Fabian mentioned here,
"However, you can query all stats that are displayed by the dashboard via a
REST API [1] while the JM is running and save them yourself. This way you
can analyze the data also after the JM was stopped" could you explain about
this sentence in detail.
I want to evaluate timeline view of each function after a job is done.
Thanks,
Phil
Re: Hello, a question about Dashborad in Flink
Posted by Stephan Ewen <se...@apache.org>.
Hi!
The REST monitoring interface and extended web dashboard were added in
version 0.10
Greetings,
Stephan
On Fri, Jan 29, 2016 at 9:55 AM, Philip Lee <ph...@gmail.com> wrote:
> Great,
>
> you menat the difference between narrow shuffle and global shuffle?
>
> I use Flink version 0.9,
> but it did not not work to access REST interface when I use "ssh tunnel"
> to remote server.
>
> it is from version of probelm?
>
> Best,
> Phil
>
>
>
> On Fri, Jan 29, 2016 at 9:46 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> The REST interface does also provide metrics about the number of records
>> and the size of the input and output of all tasks.
>> See:
>> - /jobs/<jobid>/vertices/<vertexid>
>> -
>> /jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt>
>> in
>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html#details-of-a-running-or-completed-job
>>
>> However, not all of this data is going over the network because some
>> tasks can be locally connected.
>>
>> Best, Fabian
>>
>> 2016-01-29 8:50 GMT+01:00 Philip Lee <ph...@gmail.com>:
>>
>>> Thanks,
>>>
>>> Is there any way to measure shuffle data (read and write) on Flink or
>>> Dashboard?
>>>
>>> I did not find the network usage metric in it.
>>>
>>> Best,
>>> Phil
>>>
>>> On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske <fh...@gmail.com>
>>> wrote:
>>>
>>>> You can start a job and then periodically request and store information
>>>> about the running job and vertices from using corresponding REST calls [1].
>>>> The data will be in JSON format.
>>>> After the job finished, you can stop requesting data.
>>>>
>>>> Next you parse the JSON, extract the information you need and give it
>>>> to some plotting library.
>>>> As I said, it is not possible to pass this data back into Flink's
>>>> dashboard, but you have to process and plot it yourself.
>>>>
>>>> Best, Fabian
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs
>>>>
>>>>
>>>>
>>>> 2016-01-25 16:15 GMT+01:00 Philip Lee <ph...@gmail.com>:
>>>>
>>>>> Hello,
>>>>>
>>>>> According to
>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
>>>>> I cannot retrieve the job history from Dashboard after turnning off JM.
>>>>>
>>>>> But as Fabian mentioned here,
>>>>> "However, you can query all stats that are displayed by the dashboard
>>>>> via a REST API [1] while the JM is running and save them yourself. This way
>>>>> you can analyze the data also after the JM was stopped" could you explain
>>>>> about this sentence in detail.
>>>>>
>>>>> I want to evaluate timeline view of each function after a job is done.
>>>>>
>>>>> Thanks,
>>>>> Phil
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Hello, a question about Dashborad in Flink
Posted by Philip Lee <ph...@gmail.com>.
Great,
you menat the difference between narrow shuffle and global shuffle?
I use Flink version 0.9,
but it did not not work to access REST interface when I use "ssh tunnel" to
remote server.
it is from version of probelm?
Best,
Phil
On Fri, Jan 29, 2016 at 9:46 AM, Fabian Hueske <fh...@gmail.com> wrote:
> The REST interface does also provide metrics about the number of records
> and the size of the input and output of all tasks.
> See:
> - /jobs/<jobid>/vertices/<vertexid>
> -
> /jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt>
> in
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html#details-of-a-running-or-completed-job
>
> However, not all of this data is going over the network because some tasks
> can be locally connected.
>
> Best, Fabian
>
> 2016-01-29 8:50 GMT+01:00 Philip Lee <ph...@gmail.com>:
>
>> Thanks,
>>
>> Is there any way to measure shuffle data (read and write) on Flink or
>> Dashboard?
>>
>> I did not find the network usage metric in it.
>>
>> Best,
>> Phil
>>
>> On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>>> You can start a job and then periodically request and store information
>>> about the running job and vertices from using corresponding REST calls [1].
>>> The data will be in JSON format.
>>> After the job finished, you can stop requesting data.
>>>
>>> Next you parse the JSON, extract the information you need and give it to
>>> some plotting library.
>>> As I said, it is not possible to pass this data back into Flink's
>>> dashboard, but you have to process and plot it yourself.
>>>
>>> Best, Fabian
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs
>>>
>>>
>>>
>>> 2016-01-25 16:15 GMT+01:00 Philip Lee <ph...@gmail.com>:
>>>
>>>> Hello,
>>>>
>>>> According to
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
>>>> I cannot retrieve the job history from Dashboard after turnning off JM.
>>>>
>>>> But as Fabian mentioned here,
>>>> "However, you can query all stats that are displayed by the dashboard
>>>> via a REST API [1] while the JM is running and save them yourself. This way
>>>> you can analyze the data also after the JM was stopped" could you explain
>>>> about this sentence in detail.
>>>>
>>>> I want to evaluate timeline view of each function after a job is done.
>>>>
>>>> Thanks,
>>>> Phil
>>>>
>>>
>>>
>>
>
Re: Hello, a question about Dashborad in Flink
Posted by Fabian Hueske <fh...@gmail.com>.
The REST interface does also provide metrics about the number of records
and the size of the input and output of all tasks.
See:
- /jobs/<jobid>/vertices/<vertexid>
- /jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt>
in
https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html#details-of-a-running-or-completed-job
However, not all of this data is going over the network because some tasks
can be locally connected.
Best, Fabian
2016-01-29 8:50 GMT+01:00 Philip Lee <ph...@gmail.com>:
> Thanks,
>
> Is there any way to measure shuffle data (read and write) on Flink or
> Dashboard?
>
> I did not find the network usage metric in it.
>
> Best,
> Phil
>
> On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> You can start a job and then periodically request and store information
>> about the running job and vertices from using corresponding REST calls [1].
>> The data will be in JSON format.
>> After the job finished, you can stop requesting data.
>>
>> Next you parse the JSON, extract the information you need and give it to
>> some plotting library.
>> As I said, it is not possible to pass this data back into Flink's
>> dashboard, but you have to process and plot it yourself.
>>
>> Best, Fabian
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs
>>
>>
>>
>> 2016-01-25 16:15 GMT+01:00 Philip Lee <ph...@gmail.com>:
>>
>>> Hello,
>>>
>>> According to
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
>>> I cannot retrieve the job history from Dashboard after turnning off JM.
>>>
>>> But as Fabian mentioned here,
>>> "However, you can query all stats that are displayed by the dashboard
>>> via a REST API [1] while the JM is running and save them yourself. This way
>>> you can analyze the data also after the JM was stopped" could you explain
>>> about this sentence in detail.
>>>
>>> I want to evaluate timeline view of each function after a job is done.
>>>
>>> Thanks,
>>> Phil
>>>
>>
>>
>
Re: Hello, a question about Dashborad in Flink
Posted by Philip Lee <ph...@gmail.com>.
Thanks,
Is there any way to measure shuffle data (read and write) on Flink or
Dashboard?
I did not find the network usage metric in it.
Best,
Phil
On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske <fh...@gmail.com> wrote:
> You can start a job and then periodically request and store information
> about the running job and vertices from using corresponding REST calls [1].
> The data will be in JSON format.
> After the job finished, you can stop requesting data.
>
> Next you parse the JSON, extract the information you need and give it to
> some plotting library.
> As I said, it is not possible to pass this data back into Flink's
> dashboard, but you have to process and plot it yourself.
>
> Best, Fabian
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs
>
>
>
> 2016-01-25 16:15 GMT+01:00 Philip Lee <ph...@gmail.com>:
>
>> Hello,
>>
>> According to
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
>> I cannot retrieve the job history from Dashboard after turnning off JM.
>>
>> But as Fabian mentioned here,
>> "However, you can query all stats that are displayed by the dashboard via
>> a REST API [1] while the JM is running and save them yourself. This way you
>> can analyze the data also after the JM was stopped" could you explain about
>> this sentence in detail.
>>
>> I want to evaluate timeline view of each function after a job is done.
>>
>> Thanks,
>> Phil
>>
>
>
Re: Hello, a question about Dashborad in Flink
Posted by Fabian Hueske <fh...@gmail.com>.
You can start a job and then periodically request and store information
about the running job and vertices from using corresponding REST calls [1].
The data will be in JSON format.
After the job finished, you can stop requesting data.
Next you parse the JSON, extract the information you need and give it to
some plotting library.
As I said, it is not possible to pass this data back into Flink's
dashboard, but you have to process and plot it yourself.
Best, Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs
2016-01-25 16:15 GMT+01:00 Philip Lee <ph...@gmail.com>:
> Hello,
>
> According to
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
> I cannot retrieve the job history from Dashboard after turnning off JM.
>
> But as Fabian mentioned here,
> "However, you can query all stats that are displayed by the dashboard via
> a REST API [1] while the JM is running and save them yourself. This way you
> can analyze the data also after the JM was stopped" could you explain about
> this sentence in detail.
>
> I want to evaluate timeline view of each function after a job is done.
>
> Thanks,
> Phil
>