[jira] [Comment Edited] (FLINK-3302) Save Taskmanager Metrics per Job

["Pieter Hameete (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123857#comment-15123857 ] 

Pieter Hameete edited comment on FLINK-3302 at 1/29/16 5:44 PM:
----------------------------------------------------------------

Hi Greg,

nice, I did not see your ticket, but good to see that more people would like such functionality :-) We could close this ticket and continue in yours.

My idea was to indead go for the simpler option by adding this functionality to the JobManager and re-using the heartbeat interval. I personally do not needed the advantages of logging on the TaskManager but I agree that it is better to do it right in 1 go. A best solution would be a combination though;

* Perform logging on the TaskManagers at freely choosable intervals
* Write logs accross the TaskManagers
* Guaranteed logging
* At the end of a Job aggregate the results at the JobManager for convenience. A different mechanism for this could work as well (for example being able to request the metrics for Jobs in the history section), as long as you can get the metrics per TaskManager per job in a convenient way.


was (Author: phameete):

Hi Greg,

nice, I did not see your ticket, but good to see that more people would like such functionality :-) We could close this ticket and continue in yours.

My idea was to indead go for the simpler option by adding this functionality to the JobManager and re-using the heartbeat interval. I personally do not needed the advantages of logging on the TaskManager but I agree that it is better to do it right in 1 go. A best solution would be a combination though;

* Perform logging on the TaskManagers at freely choosable intervals
* Write logs accross the TaskManagers
* Guaranteed logging
* At the end of a Job aggregate the results at the JobManager for convenience. A different mechanism for this could work as well (for example being able to request the metrics for Jobs in the history section), as long as you can get the metrics per TaskManager per job.

> Save Taskmanager Metrics per Job
> --------------------------------
>
>                 Key: FLINK-3302
>                 URL: https://issues.apache.org/jira/browse/FLINK-3302
>             Project: Flink
>          Issue Type: Wish
>          Components: JobManager, Webfrontend
>            Reporter: Pieter Hameete
>            Priority: Minor
>
> As followup to my email in the user mailing list:
> It would be nice if there is a configuration option to have the JobManager save the received metrics for each of the TaskManagers per separate job. This information can be useful for later analysis of job performance.
> The Metrics are currently already received by the JobManager and shown/updated in the Web UI every few seconds. It should not be very difficult to write these timestamped updates to disk and group per job.
> This is something I could pick up myself in a couple of months.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)