You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2016/05/12 16:00:15 UTC
[jira] [Commented] (HADOOP-13050) Upgrade to AWS SDK 10.10+ for
Java 8u60+
[ https://issues.apache.org/jira/browse/HADOOP-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281647#comment-15281647 ]
Steve Loughran commented on HADOOP-13050:
-----------------------------------------
I'll make this patch something where you can set the aws version on the command line; that way people can build hadoop with their own choice of version without having to edit the poms.
Dependencies of the 1.10.77 full SDK below
* there's a new jackson dependency, {{jackson-dataformat-cbor}} not used anywhere in hadoop —so not under hadoop's version management. It' needs to be explicitly declared in hadoop-project and then depended on in hadoop-aws.
* yes, a new joda-time
* the httpcomponents are as 10.6 as before. Meaning HADOOP-12767 exists in Hadoop 2.8 today
{code}
[INFO] +- com.amazonaws:aws-java-sdk:jar:1.10.77:compile
[INFO] | +- com.amazonaws:aws-java-sdk-support:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-simpledb:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-simpleworkflow:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-storagegateway:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-route53:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-s3:jar:1.10.77:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-kms:jar:1.10.77:compile - omitted for duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-importexport:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-sts:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-sqs:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-rds:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-redshift:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-elasticbeanstalk:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-glacier:jar:1.10.77:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-sqs:jar:1.10.77:compile - omitted for duplicate)
[INFO] | | +- (com.amazonaws:aws-java-sdk-sns:jar:1.10.77:compile - omitted for duplicate)
[INFO] | | +- (com.amazonaws:aws-java-sdk-s3:jar:1.10.77:compile - omitted for duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-iam:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-datapipeline:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-elasticloadbalancing:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-emr:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-elasticache:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-elastictranscoder:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-ec2:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-dynamodb:jar:1.10.77:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-s3:jar:1.10.77:compile - omitted for duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-sns:jar:1.10.77:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-sqs:jar:1.10.77:compile - omitted for duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudtrail:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudwatch:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-logs:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-events:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cognitoidentity:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cognitosync:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-directconnect:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudformation:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudfront:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-kinesis:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-opsworks:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-ses:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-autoscaling:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudsearch:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudwatchmetrics:jar:1.10.77:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-cloudwatch:jar:1.10.77:compile - omitted for duplicate)
[INFO] | | +- (com.amazonaws:aws-java-sdk-dynamodb:jar:1.10.77:compile - omitted for duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-swf-libraries:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-simpleworkflow:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-codedeploy:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-codepipeline:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-kms:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-config:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-lambda:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-ecs:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-ecr:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudhsm:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-ssm:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-workspaces:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-machinelearning:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-directory:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-efs:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-codecommit:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-devicefarm:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-elasticsearch:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-waf:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-marketplacecommerceanalytics:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-inspector:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-iot:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-api-gateway:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-acm:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-gamelift:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-dms:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-marketplacemeteringservice:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cognitoidp:jar:1.10.77:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile - omitted for duplicate)
[INFO] | \- com.amazonaws:aws-java-sdk-core:jar:1.10.77:compile
[INFO] | +- (commons-logging:commons-logging:jar:1.1.3:compile - version managed from 1.1.1; omitted for duplicate)
[INFO] | +- (org.apache.httpcomponents:httpclient:jar:4.2.5:compile - version managed from 4.3.6; omitted for duplicate)
[INFO] | +- (com.fasterxml.jackson.core:jackson-databind:jar:2.2.3:compile - version managed from 2.5.3; omitted for duplicate)
[INFO] | +- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.5.3:compile
[INFO] | | \- (com.fasterxml.jackson.core:jackson-core:jar:2.2.3:compile - version managed from 2.5.3; omitted for duplicate)
[INFO] | \- joda-time:joda-time:jar:2.8.1:compile
{code}
> Upgrade to AWS SDK 10.10+ for Java 8u60+
> ----------------------------------------
>
> Key: HADOOP-13050
> URL: https://issues.apache.org/jira/browse/HADOOP-13050
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
> Affects Versions: 2.7.2
> Reporter: Steve Loughran
>
> HADOOP-13044 highlights that AWS SDK 10.6 —shipping in Hadoop 2.7+, doesn't work on open jdk >= 8u60, because a change in the JDK broke the version of Joda time that AWS uses.
> Fix, update the JDK. Though, that implies updating http components: HADOOP-12767.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org