[jira] [Commented] (MAPREDUCE-6480) archive-logs tool may miss applications

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791482#comment-14791482 ] 

Hadoop QA commented on MAPREDUCE-6480:
--------------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  17m 42s | Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 1 new or modified test files. |
| {color:green}+1{color} | javac |   8m 46s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |  11m 17s | There were no new javadoc warning messages. |
| {color:green}+1{color} | release audit |   0m 25s | The applied patch does not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   0m 21s | The applied patch generated  7 new checkstyle issues (total was 4, now 11). |
| {color:green}+1{color} | whitespace |   0m  1s | The patch has no lines that end in whitespace. |
| {color:green}+1{color} | install |   1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 37s | The patch built with eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   0m 51s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | tools/hadoop tests |   0m 30s | Tests failed in hadoop-archive-logs. |
| | |  42m 12s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.tools.TestHadoopArchiveLogsRunner |
|   | hadoop.tools.TestHadoopArchiveLogs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12756389/MAPREDUCE-6480.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0832b38 |
| Pre-patch Findbugs warnings | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/artifact/patchprocess/trunkFindbugsWarningshadoop-archive-logs.html |
| checkstyle |  https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/artifact/patchprocess/diffcheckstylehadoop-archive-logs.txt |
| hadoop-archive-logs test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/artifact/patchprocess/testrun_hadoop-archive-logs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5997/console |


This message was automatically generated.

> archive-logs tool may miss applications
> ---------------------------------------
>
>                 Key: MAPREDUCE-6480
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6480
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.8.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: MAPREDUCE-6480.001.patch
>
>
> MAPREDUCE-6415 added a tool to archive aggregated logs into HAR files.  It seeds the initial list of applications to process based on apps which have finished aggregated, according to the RM.  However, the RM doesn't remember completed applications forever (e.g. failover), so it's possible for the tool to miss applications if they're no longer in the RM.  
> Instead, we should do the following:
> # Seed the initial list of apps based on the aggregated log directories
> # Make the RM not consider applications "complete" until their log aggregation has reached a terminal state (i.e. DISABLED, SUCCEEDED, FAILED, TIME_OUT).  
> #2 will allow #1 to assume that any apps not found in the RM are done aggregating.  #1 on it's own should cover most cases though



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)