[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear

["Aljoscha Krettek (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aljoscha Krettek updated FLINK-9441:
------------------------------------
    Fix Version/s: 1.7.0

> Hadoop Required Dependency List Not Clear
> -----------------------------------------
>
>                 Key: FLINK-9441
>                 URL: https://issues.apache.org/jira/browse/FLINK-9441
>             Project: Flink
>          Issue Type: Bug
>          Components: Build System, Configuration, Documentation
>    Affects Versions: 1.5.0, 1.4.2, 1.6.0
>            Reporter: Razvan
>            Priority: Blocker
>             Fix For: 1.7.0
>
>
> To be able to use Apache Flink with S3, a few libraries from Hadoop distribution are required to be added to lib/ folder.
> This list is partially documented in [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html]  (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas Flink supports 2.8 also.
> How to compose the dependecy list for Hadoop 2.8?
> Is it possible to bundle the dependencies in a separate archive that users can download?
>  
> UPDATE:
> Downloaded Apache Flink 1.4.2 for Hadoop 2.7 and it seems it was compiled with Hadoop 2.8. I get the error here: 
>  
> {{"java.lang.NumberFormatException: For input string: "100M" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:589) at java.lang.Long.parseLong(Long.java:631) at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1319) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:248) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) }}
> {{..."}}
> [https://stackoverflow.com/questions/48149929/hive-1-2-metastore-service-doesnt-start-after-configuring-it-to-s3-storage-inst?rq=1]
>  
> So I cannot start the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)