You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Bharat Viswanadham (JIRA)" <ji...@apache.org> on 2018/11/13 00:20:00 UTC
[jira] [Updated] (HADOOP-15924) Hadoop aws cannot be used with
shaded jars
[ https://issues.apache.org/jira/browse/HADOOP-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bharat Viswanadham updated HADOOP-15924:
----------------------------------------
Description:
Issue is hadoop-aws cannot be used with shaded jars.
The recommended client side jars for hadoop 3 are client-api/runtime shaded jars.
They shade guava etc. So something like SemaphoredDelegatingExecutor refers to shaded guava classes.
hadoop-aws has S3AFileSystem implementation which refers to SemaphoredDelegatingExecutor with unshaded guava ListeningService in the constructor. When S3AFileSystem is created then it uses the hadoop-api jar and finds SemaphoredDelegatingExecutor but not the right constructor because in client-api jar SemaphoredDelegatingExecutor constructor has the shaded guava ListenerService.
So essentially none of the aws/azure/adl hadoop FS implementations will work with the shaded Hadoop client runtime jars.
This Jira is created to track the work required to make hadoop-aws work with hadoop shaded client jars.
The solution for this can be, hadoop-aws depends on hadoop shaded jars. In this way, we shall not see the issue. Currently, hadoop-aws depends on aws-sdk-bundle and all other remaining jars are provided dependencies.
cc [~steve_l]
was:
Issue is hadoop-aws cannot be used with shaded jars.
The recommended client side jars for hadoop 3 are client-api/runtime shaded jars.
They shade guava etc. So something like SemaphoredDelegatingExecutor refers to shaded guava classes.
hadoop-aws has S3AFileSystem implementation which refers to SemaphoredDelegatingExecutor with unshaded guava ListeningService in the constructor. When S3AFileSystem is created then it uses the hadoop-api jar and finds SemaphoredDelegatingExecutor but not the right constructor because in client-api jar SemaphoredDelegatingExecutor constructor has the shaded guava ListenerService.
So essentially none of the aws/azure/adl hadoop FS implementations will work with the shaded Hadoop client runtime jars.
This Jira is created to track the work required to make hadoop-aws work with hadoop shaded client jars.
The solution for this can be, hadoop-aws depends on hadoop shaded jars. In this way, we shall not see the issue. Currently, hadoop-aws depends on aws-sdk-bundle and all other remaining jars are provided dependencies.
> Hadoop aws cannot be used with shaded jars
> ------------------------------------------
>
> Key: HADOOP-15924
> URL: https://issues.apache.org/jira/browse/HADOOP-15924
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Bharat Viswanadham
> Priority: Major
> Attachments: HADOOP-15924.00.patch
>
>
> Issue is hadoop-aws cannot be used with shaded jars.
> The recommended client side jars for hadoop 3 are client-api/runtime shaded jars.
> They shade guava etc. So something like SemaphoredDelegatingExecutor refers to shaded guava classes.
> hadoop-aws has S3AFileSystem implementation which refers to SemaphoredDelegatingExecutor with unshaded guava ListeningService in the constructor. When S3AFileSystem is created then it uses the hadoop-api jar and finds SemaphoredDelegatingExecutor but not the right constructor because in client-api jar SemaphoredDelegatingExecutor constructor has the shaded guava ListenerService.
> So essentially none of the aws/azure/adl hadoop FS implementations will work with the shaded Hadoop client runtime jars.
>
> This Jira is created to track the work required to make hadoop-aws work with hadoop shaded client jars.
>
> The solution for this can be, hadoop-aws depends on hadoop shaded jars. In this way, we shall not see the issue. Currently, hadoop-aws depends on aws-sdk-bundle and all other remaining jars are provided dependencies.
>
> cc [~steve_l]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org