[jira] [Created] (FLUME-3174) HdfsSink AWS S3A authentication does not work on JDK 8

["Marcell Hegedus (JIRA)" <ji...@apache.org>]

Marcell Hegedus created FLUME-3174:
--------------------------------------

             Summary: HdfsSink AWS S3A authentication does not work on JDK 8
                 Key: FLUME-3174
                 URL: https://issues.apache.org/jira/browse/FLUME-3174
             Project: Flume
          Issue Type: Bug
            Reporter: Marcell Hegedus
            Priority: Minor


Flume writing to S3A with the following Hdfs Sink configuration fails with AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden...
{code}
a1.sinks = k1
a1.sinks.k1.channel = c1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = s3a://testflume/logs
{code}

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are provided either in flume-env or in core-site.xml and running "hdfs dfs -ls s3a://testflume/logs" works properly.

The cause and the fix is documented in [hadoop-aws/index.md|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#authentication-failures-when-running-on-java-8u60]
{quote}A change in the Java 8 JVM broke some of the toString() string generation of Joda Time 2.8.0, which stopped the Amazon S3 client from being able to generate authentication headers suitable for validation by S3.

Fix: Make sure that the version of Joda Time is 2.8.1 or later, or use a new version of Java 8.{quote}

Tested that authentication is successful with
* JDK 7
* JDK 8 + joda-time updated to v2.9.6.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)