[jira] [Commented] (TEZ-2628) History logging plugin to write ATS events to HDFS

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695907#comment-14695907 ] 

Hitesh Shah commented on TEZ-2628:
----------------------------------

Did an initial test of this patch and it seems largely functional when testing just native tez jobs.  Had a couple of comments though on the actual entity logger implementation:

   - It seems to me that all the code in EntityLogger relies on YARN configs and there does not seem anything tez specific. i.e. could all of this just be wrapped in a different impl of TimelineClient itself? 
   - Additionally, this impl works well for a yarn application writing data. How do you see say the Hive client writing data to Timeline via HDFS? Would that be feasible or is the assumption that the existing write path via the Timeline webservices continue to work as long as all entities being written to via webservices are configured in the list of summary entity types?  

> History logging plugin to write ATS events to HDFS
> --------------------------------------------------
>
>                 Key: TEZ-2628
>                 URL: https://issues.apache.org/jira/browse/TEZ-2628
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: TEZ-2628.001.patch
>
>
> This provides another history logging alternative that conceptually the same as the timeline logging service but logs the entities to a file rather than posting the events to the timeline server directly.  When coupled with the timeline store plugin from YARN-3942 it allows the Tez job to be decoupled from the timeline server yet the Tez UI can still function properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)