[jira] [Created] (YARN-10940) Fix Documentation for AWS-Hadoop integration / yarn-site.xml

["Girish Ganesan (Jira)" <ji...@apache.org>]

Girish Ganesan created YARN-10940:
-------------------------------------

             Summary: Fix Documentation for AWS-Hadoop integration / yarn-site.xml
                 Key: YARN-10940
                 URL: https://issues.apache.org/jira/browse/YARN-10940
             Project: Hadoop YARN
          Issue Type: Task
            Reporter: Girish Ganesan


The following document on AWS-Hadoop integration specified authenticating via AWS environment variables:
[https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Authenticating_via_the_AWS_Environment_Variables]

It provides a warning:

_Important: These environment variables are generally not propagated from client to server when YARN applications are launched. That is: having the AWS environment variables set when an application is launched will not permit the launched application to access S3 resources. The environment variables must (somehow) be set on the hosts/processes where the work is executed._

This is somewhat cryptic. A few things need to be clarified in the doc:
 # This is true even when Yarn is running on a single node (pseudo distributed).
 # *This also affects authentication via named profile* credentials:[https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Using_Named_Profile_Credentials_with_ProfileCredentialsProvider] __ This method depends on AWS_PROFILE variable.
 # Please give some pointers on how the variables can be propagated. One way is to whitelist the variable in yarn.nodemanager.env-whitelist (set in yarn-site.xml): [https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuring_Environment_of_Hadoop_Daemons]

I was trying to figure out why hive was failing on a query (using mapred) on an external table created from S3. After a while I realized it was not getting the AWS_PROFILE variable. Eventually I realized that adding the variable to the Yarn whitelist will do the trick. Hopefully this ticket will help someone else.  

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org