[jira] [Commented] (MAPREDUCE-6638) Jobs with encrypted spills don't recover if the AM goes down

["Karthik Kambatla (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15469707#comment-15469707 ] 

Karthik Kambatla commented on MAPREDUCE-6638:
---------------------------------------------

We should have two JIRAs for this - (1) Avoid recovering an AM if encrypted spill is enabled, and (2) Support recovering an AM even when encrypted spill is enabled. I am fine with using this JIRA for either, but we should file the other one too. 

Comments on the patch:
# The comment for the variable can be simplified to be clear. For instance, "When intermediate data is encrypted, recovering the job requires access to the key. Until the encryption key is persisted, we should avoid recovery attempts."
# Can the boolean condition be simplified to {{numReduceTasks <= 0 || shuffleKeyValidForRecovery || !spillEncrypted}}? 
# I assume the new method {{recovered()}} is for tests only. Should we annotate it @VisibleForTesting? 


> Jobs with encrypted spills don't recover if the AM goes down
> ------------------------------------------------------------
>
>                 Key: MAPREDUCE-6638
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 2.7.2
>            Reporter: Karthik Kambatla
>            Assignee: Haibo Chen
>            Priority: Critical
>         Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch
>
>
> Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be recovered if the AM fails. We should store the key some place safe so they can actually be recovered. If there is no "safe" place, at least we should restart the job by re-running all mappers/reducers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org