[jira] [Commented] (BIGTOP-4548) Accessing S3A file system fails because of missing AWS SDK

["Masatake Iwasaki (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097043#comment-18097043 ] 

Masatake Iwasaki commented on BIGTOP-4548:
------------------------------------------

I'm not so positive to redistribute the AWS SDK bundle as a part of Bigtop packages due to its size. Adding (optional) Puppet manifest for downloading AWS SDK bundle and [setting HADOOP_OPTIONAL_TOOLS|https://hadoop.apache.org/docs/r3.4.3/hadoop-aws/tools/hadoop-aws/index.html#Overview] would work. We can mention [the release notes of Hadoop 3.4.3 describing the drop of AWS SDK bundle|https://hadoop.apache.org/release/3.4.3.html] at least for Bigtop 3.6.0.

> Accessing S3A file system fails because of missing AWS SDK
> ----------------------------------------------------------
>
>                 Key: BIGTOP-4548
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-4548
>             Project: Bigtop
>          Issue Type: Bug
>          Components: hadoop
>    Affects Versions: 3.6.0
>         Environment: Ubuntu 24.04, ARM64, AWS
>            Reporter: Sebastian Nagel
>            Priority: Major
>
> (seen while testing Bigtop 3.6.0 RC0 on Ubuntu 24.04, ARM64, AWS EC2)
> On Bigtop 3.5.0 the hadoop-aws.jar and the required AWS SDK are contained in the package "hadoop" and are installed into the tools lib dir:
> {noformat}
>  /usr/lib/hadoop/tools/lib/hadoop-aws-3.3.6.jar
>  /usr/lib/hadoop/tools/lib/aws-java-sdk-bundle-1.12.367.jar
> {noformat}
> On Bigtop 3.6.0 (RC0)
> - the hadoop-aws jar is moved into the lib dir:
> {noformat}
>  /usr/lib/hadoop/lib/hadoop-aws-3.4.3.jar
> {noformat}
> - the AWS SDK bundle is missing
> Access to S3 via S3A fails:
> {noformat}
> $> hadoop fs -ls s3a://mybucket/
> Exception in thread "main" java.lang.NoClassDefFoundError: software/amazon/awssdk/core/exception/SdkException
> {noformat}
> Related issues:
> - HADOOP-19083 removed the AWS SDK from the binary distribution because of its large size.
> - HADOOP-19696 has moved the cloud connectors to the common lib dir to avoid extra setup when accessing data on cloud storage.
>   - See BIGTOP-3495 which moved the cloud packages to the tools lib dir.
>   - hadoop-aws.sh in the Hadoop binary tarball also still references the tools lib dir (that's a Hadoop issue, not a Bigtop one)
> The problem can be easily worked around by
> - Determine the correct version used with Hadoop 3.4.3:
>   - see <https://github.com/apache/hadoop/blob/branch-3.4.3/hadoop-project/pom.xml#L207>
>   - => 2.35.4
> - Download and install the AWS SDK bundle:
> {noformat}
> mvn dependency:get -Dartifact=software.amazon.awssdk:bundle:2.35.4
> sudo cp ~/.m2/repository/software/amazon/awssdk/bundle/2.35.4/bundle-2.35.4.jar /usr/lib/hadoop/lib/
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)