You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Obaid Salikeen <Ob...@iacpublishinglabs.com> on 2016/01/06 23:00:55 UTC
AWS S3 Flume Sink issue
Hi All,
I am trying to use Flume (1.6.0) to gather events from Kafka and publish them on HDFS and S3. I am experiencing errors whenever I try to use S3 Sink. It seems like HDFS Sink does not recognize S3 filesystem, or I may be missing a configuration (Link I used for help: http://stackoverflow.com/questions/26028096/use-flume-to-stream-data-to-s3).
1. Configuration File:
# Sinks
# 2. S3
etl.sinks.s3-sink.type = hdfs
etl.sinks.s3-sink.hdfs.path = s3n://XXXX:YYYY@myBucket/dev/
etl.sinks.s3-sink.hdfs.fileType = DataStream
2. Following is the error I get:
06 Jan 2016 13:00:40,941 WARN [hdfs-hdfs-sink-call-runner-0] (org.apache.hadoop.util.NativeCodeLoader.<clinit>:62) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
06 Jan 2016 13:00:41,093 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.open:234) - Creating s3://XXX:YYY@dp-unified-etl//FlumeData.1452114040235.tmp
06 Jan 2016 13:00:41,121 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:455) - HDFS IO error
java.io.IOException: No FileSystem for scheme: s3
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Any help to resolve this issue will be highly appreciated !
Thanks
Obaid