[jira] [Updated] (OOZIE-562) Design of Oozie Logging System

["Virag Kothari (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OOZIE-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Virag Kothari updated OOZIE-562:
--------------------------------

    Assignee: Kiran Nagasubramanian
    
> Design of Oozie Logging System
> ------------------------------
>
>                 Key: OOZIE-562
>                 URL: https://issues.apache.org/jira/browse/OOZIE-562
>             Project: Oozie
>          Issue Type: Wish
>            Reporter: Kiran Nagasubramanian
>            Assignee: Kiran Nagasubramanian
>
> When large log files are there in the log folder, even if we try to retrieve the log content(of size of the order of kBs) for a small job, it takes quite some significant amount of time, as equally as it takes to retrieve log content for very large jobs(of the order of GBs). This happens because, the list of files to be scanned for log retrieval is the same for all jobs that run approximately at around the same time.
> Chances that this might materialize in production systems is really high. Since, hundreds of jobs would be logging to the same file for an hr and this file size would be really huge. Is it possible to have the logs for the jobs separately so that scanning large log files of other jobs can be avoided? Would this be really worth the effort?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira