[jira] [Commented] (HADOOP-11528) Document precedence rules for generic command line options.

["Chris Nauroth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-11528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299029#comment-14299029 ] 

Chris Nauroth commented on HADOOP-11528:
----------------------------------------

This came out of discussion in HADOOP-11509:

https://issues.apache.org/jira/browse/HADOOP-11509?focusedCommentId=14293817&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14293817

The generic command line options are documented in two places:

http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options

http://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopStreaming.html#Generic_Command_Options

There is nothing that makes it clear to the end user that certain arguments take precedence over other arguments.  The current order of precedence, as implemented in {{GenericOptionsParser#processGeneralOptions}} is:
# -fs
# -jt
# -conf
# -D
# -tokenCacheFile (which isn't even mentioned in the current docs)

Items lower in this list overwrite ("take precedence") over items higher in the list in the final resulting {{Configuration}}.  For example, suppose a command line invocation passes both -fs and -D fs.defaultFS.  In this example, the -D value wins, because it has higher precedence.  That's the value you'll get in the final {{Configuration}}.

Note that the precedence rules don't really apply to the -libjars, -files and -archives options.  That's because these options don't directly map to a single configuration property, so there really is no opportunity for higher precedence things to overwrite them.

> Document precedence rules for generic command line options.
> -----------------------------------------------------------
>
>                 Key: HADOOP-11528
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11528
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Chris Nauroth
>            Priority: Minor
>
> Our documentation does not clearly the state the precedence rules when a command line invocation uses multiple generic command line options.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)