You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Obaid Salikeen <Ob...@iacpublishinglabs.com> on 2016/01/06 23:00:55 UTC

AWS S3 Flume Sink issue

Hi All,

I am trying to use Flume (1.6.0) to gather events from Kafka and publish them on HDFS and S3. I am experiencing errors whenever I try to use S3 Sink. It seems like HDFS Sink does not recognize S3 filesystem, or I may be missing a configuration (Link I used for help: http://stackoverflow.com/questions/26028096/use-flume-to-stream-data-to-s3).


1.       Configuration File:
# Sinks
# 2. S3
etl.sinks.s3-sink.type = hdfs
etl.sinks.s3-sink.hdfs.path = s3n://XXXX:YYYY@myBucket/dev/
etl.sinks.s3-sink.hdfs.fileType = DataStream



2.       Following is the error I get:
06 Jan 2016 13:00:40,941 WARN  [hdfs-hdfs-sink-call-runner-0] (org.apache.hadoop.util.NativeCodeLoader.<clinit>:62)  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
06 Jan 2016 13:00:41,093 INFO  [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.open:234)  - Creating s3://XXX:YYY@dp-unified-etl//FlumeData.1452114040235.tmp
06 Jan 2016 13:00:41,121 WARN  [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:455)  - HDFS IO error
java.io.IOException: No FileSystem for scheme: s3
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Any help to resolve this issue will be highly appreciated !

Thanks
Obaid