Having issues with snappy compression

[Dinesh Narayanan <nd...@gmail.com>]

Hello,
I just started learning Flume. I get this error when i try to enable snappy
compression with Flume 1.6 on Mac OSX 10.9.5.

However, my other MR jobs with snappy compression work ok.

org.apache.flume.EventDeliveryException: java.lang.RuntimeException: native
snappy library not available: this version of libhadoop was built without
snappy support.
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: native snappy library not available:
this version of libhadoop was built without snappy support.
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1201)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1094)
at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1444)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:582)
at
org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:98)
at
org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:78)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:254)
at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
at
org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

My sample test sink configuration is
agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.hdfs.path = /metrics/raw/flume/%Y/%m/%d/%H
agent1.sinks.sink1.hdfs.filePrefix = metrics
agent1.sinks.sink1.hdfs.fileSuffix = .seq
agent1.sinks.sink1.hdfs.fileType = SequenceFile
agent1.sinks.sink1.hdfs.writeFormat = Text
agent1.sinks.sink1.hdfs.rollInterval = 3
agent1.sinks.sink1.hdfs.rollSize = 1048576 # 1mb
agent1.sinks.sink1.hdfs.rollCount = 1000
agent1.sinks.sink1.hdfs.codeC = snappy

What could i be missing here.

Thanks
Dinesh