[GitHub] [hadoop] steveloughran opened a new pull request #685: HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true

[GitBox <gi...@apache.org>]

steveloughran opened a new pull request #685: HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true
URL: https://github.com/apache/hadoop/pull/685
 
 
   This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
   
   Production code changes:
   * S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
   
   Tests
   * Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
   * Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
   * And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
   
   Docs:
   * Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
   
   Contributed by Steve Loughran.
   
   Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org