[jira] [Commented] (HDDS-9175) Consider including hadoop-aws jar in Ozone distribution to allow s3a:// access with `ozone fs`

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-9175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757913#comment-17757913 ] 

Steve Loughran commented on HDDS-9175:
--------------------------------------

you should be aware that aws-sdk-bundle jar is *huge*, and even if you include the hadoop-aws module you need a plan for getting that out -and bundling it makes your distro a lot bigger

> Consider including hadoop-aws jar in Ozone distribution to allow s3a:// access with `ozone fs`
> ----------------------------------------------------------------------------------------------
>
>                 Key: HDDS-9175
>                 URL: https://issues.apache.org/jira/browse/HDDS-9175
>             Project: Apache Ozone
>          Issue Type: Task
>            Reporter: Siyao Meng
>            Priority: Major
>
> Right now a simple `fs -ls s3a://` fails because hadoop-aws jar is not in ozone classpath by default:
> {code}
> $ ozone fs -ls s3a://bkt11690356121/.snapshot/cm-auto-d33c7527-hourly.daily-1690356245/
> -ls: Fatal internal error
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
>     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2702)
>     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3413)
>     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3448)
>     at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:161)
>     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3556)
>     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3503)
>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:521)
>     at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> {code}
> I think it is reasonable to at least have a (optional?) build option to include hadoop-aws jar if users need it.
> Alternatively we could document this to ask users to add hadoop-aws jar to OZONE_CLASSPATH env variable when they need it.
> Thoughts? [~adoroszlai]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org