[jira] [Commented] (FLUME-2370) 'deflate' doesn't work in Flume avro sinc/source

["Shadocko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112295#comment-15112295 ] 

Shadocko commented on FLUME-2370:
---------------------------------

Hello, just to follow up on this issue: it can still happen in 2016, although there's no clear path to reproduce the problem.
The issue started for me after testing many configurations (for performance tuning purposes), so after quite a few configuration changes and restarts of both Flume agents.

The workaround was to specify a compression level at both ends (Avro Sink AND Source), even though It just used to work by setting the compression level only at the sink side, until some indeterminate point in time that is.

> 'deflate' doesn't work in Flume avro sinc/source
> ------------------------------------------------
>
>                 Key: FLUME-2370
>                 URL: https://issues.apache.org/jira/browse/FLUME-2370
>             Project: Flume
>          Issue Type: Documentation
>          Components: Docs
>    Affects Versions: v1.4.0
>            Reporter: Cesar Delgado
>         Attachments: patch-2370.diff
>
>
> I was trying to setup a basic 2-tier Flume setup using the Avro source/sink to communicate between tiers. When I setup the system I used compression-type on the 1st tier's sink side like documented here (https://flume.apache.org/FlumeUserGuide.html#avro-sink)
> agent.sinks.snk1.compression-type = deflate
> And the same on the source side:
> agent.sources.src1.compression-type = deflate
> But I was getting this error:
> 08 Apr 2014 00:15:07,606 WARN pool-7-thread-1 (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught:201) - Unexpected exception from downstream.
> org.jboss.netty.handler.codec.compression.CompressionException: decompression failure (-3): unknown compression method
> at org.jboss.netty.handler.codec.compression.ZlibUtil.exception(ZlibUtil.java:31)
> at org.jboss.netty.handler.codec.compression.ZlibUtil.fail(ZlibUtil.java:27)
> at org.jboss.netty.handler.codec.compression.ZlibDecoder.decode(ZlibDecoder.java:153)
> at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:67)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> After some digging I found if I changed 'deflate' for 'gzip' it works.
> agent.sinks.snk1.compression-type = gzip
> agent.sources.src1.compression-type = gzip



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)