[jira] [Comment Edited] (LOG4J2-1076) Flume appender fails to preform

["tzachi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LOG4J2-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617858#comment-14617858 ] 

tzachi edited comment on LOG4J2-1076 at 7/8/15 2:21 AM:
--------------------------------------------------------

Hey Ralph,

1. In order to make it compatible with Flume's Avro source, I think that a "deflate" compression method should be supported, especially when (as mentioned) the compression ratio is significantly better than the current GZIP compression.
2. I will test it against the fix for batching the events as soon as it becomes available (on v2.4).
3. Sorry for my typos, I did mean crashed, and I didn't provide any details because there was nothing in the logs besides system alerts for Disk IO > 90% when I used the Persistent mode. The Avro mode crashes might be a result of the terminal crash, so you can ignore my comment on this for now.
4. It should not take over 2 Hours to send 2G, especially when it took me 20 minutes using a local flume instance using the exact same link. I could not find any heavy load performance test results using this appender, but from my tests it looks like there are several issues with both Avro and Persistent mode when trying to use this appender on high number of events to a remote machine (I wish I could pin point it more).
5. I am well aware to the embeded variation example, but it nothing worked until I changed the configuration to the embeded agent configuration as described in the flume project https://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent. Regarding the unit test you mentioned, you can take a look at embedded.xml under the test resources for log4j2flume-ng and see that it matches the flume project documentation and not the log4j2 flume appender one.

I agree that there is not a particular bug, but I wasn't sure what is the best way to mark this issue. Let me know if you want me to split it to several tickets, change the ticket type from bug to something else or provide you with more information.


was (Author: tezra):
Hey Ralph,

1. In order to make it compatible with Flume's Avro source, I think that a "deflate" compression method should be supported, especially when (as mentioned) the compression ratio is significantly better than the current GZIP compression.
2. I will test it against the fix for batching the events as soon as it becomes available (on v2.4).
3. Sorry for my typos, I did mean crashed, and I didn't provide any details because there was nothing in the logs besides system alerts for Disk IO > 90% when I used the Persistent mode. The Avro mode crashes might be a result of the terminal crash, so you can ignore my comment on this for now.
4. It should not take over 2 Hours to send 2G, especially when it took me 20 minutes using a local flume instance using the exact same link. I could not find any heavy load performance test results using this appender, but from my tests it looks like there are several issues with both Avro and Persistent mode when trying to use this appender on high number of events to a remote machine(I wish I could pin point it more).
5. I am well aware to the embeded variation example, but it nothing worked until I changed the configuration to the embeded agent configuration as described in the flume project https://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent. Regarding the unit test you mentioned, you can take a look at embedded.xml under the test resources for log4j2flume-ng and see that it matches the flume project documentation and not the log4j2 flume appender one.

I agree that there is not a particular bug, but I wasn't sure what is the best way to mark this issue. Let me know if you want me to split it to several tickets, change the ticket type from bug to something else or provide you with more information.

> Flume appender fails to preform
> -------------------------------
>
>                 Key: LOG4J2-1076
>                 URL: https://issues.apache.org/jira/browse/LOG4J2-1076
>             Project: Log4j 2
>          Issue Type: Bug
>            Reporter: tzachi
>
> Recently I was testing log4j2-flume appender performance and thought to share the results as I believe they reflect some bugs/flaws in the current implementation. I conducted a series of in which I used the same 2.2 GB base file. I wrote a small java application that read he file line by line and log each line log4j2 with flume appender that sends it to another flume instance on a remote machine. I measured the time it took and traffic between the end points as I was mainly curious in the avro compression abilities.
> First, Log4j2 Flumes' appender support only GZIP compression (and only for the body) so first I was curious if this feature actually compatible with the flume's defalte compression method for avro. I found out that it didn't, and in order to make it work I would have to write my own gzipDecoder on the other side.
> Type Avro:
> 1. The process of sending the logs was VERY VERY long (over 2 hours), and crushed several times.
> 2. The more surprising part was that the traffic measured on the link was over 2G (when I used GZIP compression), and even closer to 3G (without compression). I am not even sure why there was such an overhead, but that’s what I saw several times.
> 3. The event were send one by one even when I defined a batch mode of 1000. After reading the code a little bit, I found out that batch mode is currently not supported and will might be possible on the next release -  https://issues.apache.org/jira/browse/LOG4J2-1044?jql=project%20%3D%20LOG4J2%20AND%20priority%20%3D%20Major%20AND%20resolution%20%3D%20Unresolved%20AND%20text%20~%20%22flume%22%20ORDER%20BY%20key%20DESC.
>  
> Type Persistent:
> 1. Crushed several times after a little while and stopped sending messages. It didn’t look like the flume instance was the one that crushes so my guess is that the BerkelyDB thread or something like that. I could not figured out what exactly.
> 2. During the same time it crushed (which seems pretty connected to the issue) I got alerts regarding IO stating Disk IO > 90%. Again, not sure why it happened, but it happened on several occasions and only when I tried the persistent type.
> 3. Batch mode though worked.
>  
> Type Embedded:
> 1. The documentation in log4j website about it does not reflect the the way to configure this type. I had to work my way through the errors until I got my code to run, and even then it didn’t really seem like it sends anything. Not sure why, and I probably need to look deeper into it.
>  
> Since Avro type is the only one that seems to work without a significant crush, I tested this mode of operation by adding a local Flume which get the data from the log4j2 appender and ship it to the remote Flume using deflate compression. Using this setup it took 1276484 ms ~ 21 Minutes.
> Another important thing I wanted to point out is once I removed all the appenders, it took only 10781 ms (about 10-11 seconds) to read the file. With file appender it took 99682 ms (about 1.5 minutes). So the performance drawback when using the flume appender seems pretty huge, but it can probably be reduced using the async logger mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org