You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2017/12/15 20:39:00 UTC
[jira] [Updated] (MAPREDUCE-6988) Let JHS support different file
systems for intermediate_done and done
[ https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated MAPREDUCE-6988:
-------------------------------------------
Fix Version/s: (was: 2.7.5)
> Let JHS support different file systems for intermediate_done and done
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-6988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6988
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
> Affects Versions: 2.7.4
> Reporter: Johan Gustavsson
> Priority: Minor
> Attachments: MAPREDUCE-6988.000.patch, MAPREDUCE-6988.001.patch, MAPREDUCE-6988.002.patch, MAPREDUCE-6988.003.patch
>
>
> Currently JHS uses filecontext to move files from intermediate_done to done folder. Since filecontext limits the use to 1 filesystem it makes it harder to use s3 as a storage for jhist files. By moving this to filesystem interface we can set hdfs for intermediate storage and s3 as long term storage therefore reducing the number of puts to s3 and removing the need for all M/R containers to carry a s3 sdk.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org