[jira] [Created] (FLINK-9244) Can't create a BucketingSink with a provided Configuration if no hadoop defaults

["Julien Cuquemelle (JIRA)" <ji...@apache.org>]

Julien Cuquemelle created FLINK-9244:
----------------------------------------

             Summary: Can't create a BucketingSink with a provided Configuration if no hadoop defaults
                 Key: FLINK-9244
                 URL: https://issues.apache.org/jira/browse/FLINK-9244
             Project: Flink
          Issue Type: Bug
          Components: Streaming Connectors
    Affects Versions: 1.4.2
            Reporter: Julien Cuquemelle


We build Integration tests using this kind of code: 
{code:java}
val bucketingSink = new BucketingSink[Row](s"hdfs:///user/${System.getenv("USER")}/application_name/") bucketingSink.setFSConfig(hadoopRule.getConfiguration.hdfs)
bucketingSink.setBucketer(new DateTimeBucketer[Row]("yyyy-MM-dd--HHmm"))
outpuStream.addSink(bucketingSink)
{code}
Here, the hadoopRule is providing a valid hdfs config that should allows this kind of code to run on a machine with no HADOOP_HOME or HADOOP_CONF_DIR set up, like a developper workstation or a Jenkins slave.

When running this code on such a machine, the .createHadoopFileSystem(...) fails with 

 
{noformat}
The given file system URI (hdfs:///user/$USER/application_name/) did not describe the authority
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401){noformat}

 because it tries to instantiate the fileSystem from a default configuration in .getUnguardedFileSystem() ; as the default conf doesn't exist, the default filesystem resolves to "file:///" and the checks of the consistency of the URI fails because no authority can be found


So the whole filesystem creation fails before actually trying to use the provided config.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)