[GitHub] [pulsar] BewareMyPower commented on issue #8254: pulsar io hdfs could not execute

[GitBox <gi...@apache.org>]

BewareMyPower commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-708165443


   See https://github.com/apache/pulsar/blob/3a298f3404d597e6a94de981c5fbe570264dcba1/pulsar-io/hdfs2/src/main/java/org/apache/pulsar/io/hdfs2/sink/HdfsAbstractSink.java#L92-L99
   
   `path` is created from your `directory` and other config params.
   
   The HDFS sink just uses a Hadoop client to create `FileSystem` and open a file under the configured `directory` to write or append. So you should not add any prefix to the path of `directory`.
   
   > if i change the configuration of directory to “/tmp/bar”, it will write the file to the local file system
   
   It means there's something wrong with your `hdfs-site.xml`, it cannot load the Hadoop file system, so it use local file system instead.
   
   You can write a simple Hadoop client example to verify if the issue is related to the `pdfs-site.xml`, like:
   
   ```java
   Configuration conf = new Configuration();
   conf.addResource(new Path(/* path of your pdfs-site.xml */));
   FileSystem fs = FileSystem.get(conf);
   FSDataOutputStream stream = fs.create(/* file path */);
   // Then use `stream` to write some data
   ```
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org