[jira] [Commented] (HADOOP-15924) Hadoop aws does not use shaded jars

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685144#comment-16685144 ] 

Steve Loughran commented on HADOOP-15924:
-----------------------------------------

ps, it's moot, but the tests failed due to the repackaging as {{MiniDFSCluster}} won't start. 

{code}
org/apache/hadoop/hdfs/server/namenode/FsImageProto$ErasureCodingSection$Builder.addPoliciesBuilder(I)Lorg/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ErasureCodingPolicyProto$Builder; @8: invokevirtual
  Reason:
    Type 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ErasureCodingPolicyProto' (current frame, stack[2]) is not assignable to 'com/google/protobuf/GeneratedMessage'
  Current Frame:
    bci: @8
    flags: { }
    locals: { 'org/apache/hadoop/hdfs/server/namenode/FsImageProto$ErasureCodingSection$Builder', integer }
    stack: { 'com/google/protobuf/RepeatedFieldBuilder', integer, 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ErasureCodingPolicyProto' }
  Bytecode:
    0x0000000: 2ab7 000e 1bb8 004a b600 4cc0 0046 b0  
{code}

> Hadoop aws does not use shaded jars
> -----------------------------------
>
>                 Key: HADOOP-15924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15924
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2.0
>            Reporter: Bharat Viswanadham
>            Priority: Major
>         Attachments: HADOOP-15924.00.patch
>
>
> Issue is hadoop-aws cannot be used with shaded jars.
> The recommended client side jars for hadoop 3 are client-api/runtime shaded jars.
> They shade guava etc. So something like SemaphoredDelegatingExecutor refers to shaded guava classes.
> hadoop-aws has S3AFileSystem implementation which refers to SemaphoredDelegatingExecutor with unshaded guava ListeningService in the constructor. When S3AFileSystem is created then it uses the hadoop-api jar and finds SemaphoredDelegatingExecutor but not the right constructor because in client-api jar SemaphoredDelegatingExecutor constructor has the shaded guava ListenerService.
> So essentially none of the aws/azure/adl hadoop FS implementations will work with the shaded Hadoop client runtime jars.
>  
> This Jira is created to track the work required to make hadoop-aws work with hadoop shaded client jars.
>  
> The solution for this can be, hadoop-aws depends on hadoop shaded jars. In this way, we shall not see the issue. Currently, hadoop-aws depends on aws-sdk-bundle and all other remaining jars are provided dependencies.
>  
> cc [~steve_l]
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org