[jira] [Updated] (TEZ-2628) History logging plugin to write ATS events to HDFS

["Jason Lowe (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Lowe updated TEZ-2628:
----------------------------
    Attachment: TEZ-2628.003.patch

We ran into a problem with the domain files and HiveServer2.  The domain files were relying on the client JVM shutting down to close the files, but HiveServer2 never exits and the files were never closing.  This caused a pileup of open files over time.

Here's an updated patch against master that changes the domain file handling so each domain put writes the data and closes the file.  This will not be ideal if there are many domain puts occurring for an application, but in practice we see only one domain put per app.

> History logging plugin to write ATS events to HDFS
> --------------------------------------------------
>
>                 Key: TEZ-2628
>                 URL: https://issues.apache.org/jira/browse/TEZ-2628
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: TEZ-2628.001.patch, TEZ-2628.002.patch, TEZ-2628.003.patch, hive-timeline.json
>
>
> This provides another history logging alternative that conceptually the same as the timeline logging service but logs the entities to a file rather than posting the events to the timeline server directly.  When coupled with the timeline store plugin from YARN-3942 it allows the Tez job to be decoupled from the timeline server yet the Tez UI can still function properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)