You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-user@logging.apache.org by SimonMo <si...@gmail.com> on 2012/10/25 15:34:49 UTC
Flume Appender
Hi,
does anybody know how stable is the flume appender embedded?
I have a network with nodes logging 20.000 logs/sec and i want to use Log4j
2 -> Flume -> Elasticsearch.
Is this a good approach?
I could use additionally a FailoverAppender to get my logs on a second route
to my server...
hmm, i'm not sure.
Has anybody experience with so much logs in an network?
Regards,
Simon
--
View this message in context: http://old.nabble.com/Flume-Appender-tp34601088p34601088.html
Sent from the Log4j - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Flume Appender
Posted by Christian Grobmeier <gr...@gmail.com>.
Simon,
log4j 2.0 including Flume Adapter is in beta. I don't know anybody who
tried it with that metric so far (I might be wrong, Ralph might have
done a perf test). Anyway, as with all beta software I recommend you
good testing before you put it into production. In any way... at
logging we all would be VERY happy to hear about your experiences with
your approach.
Cheers
Christian
On Thu, Oct 25, 2012 at 3:34 PM, SimonMo <si...@gmail.com> wrote:
>
> Hi,
>
> does anybody know how stable is the flume appender embedded?
> I have a network with nodes logging 20.000 logs/sec and i want to use Log4j
> 2 -> Flume -> Elasticsearch.
> Is this a good approach?
>
> I could use additionally a FailoverAppender to get my logs on a second route
> to my server...
> hmm, i'm not sure.
>
> Has anybody experience with so much logs in an network?
>
> Regards,
> Simon
> --
> View this message in context: http://old.nabble.com/Flume-Appender-tp34601088p34601088.html
> Sent from the Log4j - Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
--
http://www.grobmeier.de
https://www.timeandbill.de
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Flume Appender
Posted by SimonMo <si...@gmail.com>.
Thank you for your helpful answers and your great job.
I will test my own system in the next weeks and post the results in this
topic. Then i could say how good my log4j-flume-elasticsearch-idea is ;)
But is there any disadvantages of nesting? I could build:
Failover Appender -> Asynch Appender -> Flume Appender
-> Asynch Appender -> 2nd Flume Appender with
other host
So i have a big secure Appender.
Sorry for that question, but i only read in the manual about the advantages
of each appender. Why i shouldn't build such a
failvoer-async-flame-appender?
Regards,
Simon
Ralph Goers wrote:
>
> As Christian pointed out, Log4j 2 is still considered beta quality
> software. While it works fine in the environments I have tested in others
> are starting to use it in other environments and as they encounter
> problems and fixes are implemented the quality will improve.
>
> That said, I am using the Flume Appender and we recently did some
> performance testing. The chart below shows the result of a single
> application writing the specified number of events. The application and
> remote agents are running in VMs, which is not going to yield the best
> performance Flume has to offer. However, our application doesn't really
> reach a very large scale until we start aggregating events from a number
> of applications into Flume Agents acting as collectors. We don't expect a
> single application to be writing more than about 10 events per second so
> the numbers below are just fine even in the non-embedded case. The agent
> collectors can be run on bare metal and achieve a fairly high level of
> performance. In addition, you can run a cluster of agents in your
> collector pool and point the applications to different agents to scale up
> to the numbers you are talking about. Of course, your ultimate destination
> also has to support that kind of rate. I am not familiar with
> Elasticsearch. For more discussion on how to get Flume to scale I would
> suggest asking on the Flume mailing lists.
>
> As you can see, the embedded agent is significantly faster than the remote
> agent. The downside though, is that when you want to update Flume your
> application has to be modified.
>
> I had a discussion with the Flume developers just last week about the
> embedded appender. The current release of Flume doesn't support embedded
> out of the box so I had to do a bit of work. In one of the next releases
> they plan on improving the support. That really should impact the visible
> portion of the Appender but should make it easier to implement and
> maintain.
>
> HTH
> Ralph
>
>
>
>
>
> On Oct 25, 2012, at 6:34 AM, SimonMo wrote:
>
>>
>> Hi,
>>
>> does anybody know how stable is the flume appender embedded?
>> I have a network with nodes logging 20.000 logs/sec and i want to use
>> Log4j
>> 2 -> Flume -> Elasticsearch.
>> Is this a good approach?
>>
>> I could use additionally a FailoverAppender to get my logs on a second
>> route
>> to my server...
>> hmm, i'm not sure.
>>
>> Has anybody experience with so much logs in an network?
>>
>> Regards,
>> Simon
>> --
>> View this message in context:
>> http://old.nabble.com/Flume-Appender-tp34601088p34601088.html
>> Sent from the Log4j - Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>
>
>
>
--
View this message in context: http://old.nabble.com/Flume-Appender-tp34601088p34604533.html
Sent from the Log4j - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Flume Appender
Posted by Ralph Goers <rg...@apache.org>.
Just for clarification, I tested that way because we wanted to insure there would never be any data loss - which you can have with async appenders and loggers.
Sent from my iPad
> On Jul 24, 2014, at 2:59 AM, jfloviou <jf...@hotmail.com> wrote:
>
> Hi all,
>
> I'm digging out this Thread from December 2012 as i would like to get fresh
> thoughts on these questions.
>
> Before reading the discussion i was also hesitating between a Flume Embedded
> Agent or a Flume Appender after and Asynch Appender.
> Ralph performance output were very interesting but i feel like he compared
> Embedded agent vs Flume Appender without Asynch, with of course great
> advantage to the Embedded Agent.
>
> => Did anybody get to the same question and has fresh thoughts on this topic
> ?
>
> Best regards
>
> Jeff
>
>
>
> --
> View this message in context: http://apache-logging.6191.n7.nabble.com/Flume-Appender-tp7089p49684.html
> Sent from the Log4j - Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Flume Appender
Posted by jfloviou <jf...@hotmail.com>.
Hi all,
I'm digging out this Thread from December 2012 as i would like to get fresh
thoughts on these questions.
Before reading the discussion i was also hesitating between a Flume Embedded
Agent or a Flume Appender after and Asynch Appender.
Ralph performance output were very interesting but i feel like he compared
Embedded agent vs Flume Appender without Asynch, with of course great
advantage to the Embedded Agent.
=> Did anybody get to the same question and has fresh thoughts on this topic
?
Best regards
Jeff
--
View this message in context: http://apache-logging.6191.n7.nabble.com/Flume-Appender-tp7089p49684.html
Sent from the Log4j - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Flume Appender
Posted by Ralph Goers <ra...@dslextreme.com>.
One other thing I just remembered. When you are using the embedded appender you are writing to the embedded flume. In our testing we found that we were getting control back in our application long before the embedded Flume had a chance to forward all the messages to the remote agent. If the application terminates before the embedded agent completes its work some of the events will be left in the File Channel so it may appear you are losing events. We had to add a wait to the end of our application to verify all the events were delivered.
Ralph
On Dec 4, 2012, at 7:16 AM, Ralph Goers wrote:
> I will check with the QA engineer who ran our tests. However, you might want to test with Flume 1.3.0 which was just released. We actually use the CDH version of Flume from Cloudera and so our tests had many of the fixes that are in 1.3. I'm not sure what if what we were measuring is what you are. We tested the performance difference is what the application sees, not the time to make it to the repository.
>
> In our tests the remote agent is running on a remote machine where the latency is most likely the primary reason the embedded appender performs so much better.
>
> Also, since you are using a vanilla pattern layout your log events may be smaller than ours. We are using StructuredDataMessages and each of the items in the Message are added as a header to the FlumeEvent, in addition to being included in the message. This allows them to be efficiently inserted into Cassandra.
>
> Ralph
>
> On Dec 4, 2012, at 3:44 AM, SimonMo wrote:
>
>> Hi Ralph,
>>
>> can u give me your properties for this test.
>> i've got the same speed for both appenders.
>> and the second problem with the embedded: if i send 20.000 logs i only get
>> ~17.000 logs.
>>
>> embedded:
>> <Flume name="embedded" suppressExceptions="false" compress="false"
>> embedded="true" batchsize="1">
>> <Property name="channels">file</Property>
>> <Property name="channels.file.type">file</Property>
>> <Property
>> name="channels.file.checkpointDir">filechannel/checkpoint</Property>
>> <Property name="channels.file.dataDirs">filechannel/data</Property>
>> <Property name="sinks">agent1</Property>
>> <Property name="sinks.agent1.channel">file</Property>
>> <Property name="sinks.agent1.type">avro</Property>
>> <Property name="sinks.agent1.hostname">127.0.0.1</Property>
>> <Property name="sinks.agent1.port">41414</Property>
>> <PatternLayout pattern="%m" />
>> </Flume>
>>
>> not embedded:
>> <Flume name="flumeAppender" suppressExceptions="false" compress="false"
>> embedded="false">
>> <Agent host="127.0.0.1" port="42424" />
>> <PatternLayout>
>> <pattern>%m</pattern>
>> </PatternLayout>
>> </Flume>
>>
>> Regards,
>> Simon
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-logging.6191.n7.nabble.com/Flume-Appender-tp7089p35073.html
>> Sent from the Log4j - Users mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Flume Appender
Posted by Ralph Goers <ra...@dslextreme.com>.
I will check with the QA engineer who ran our tests. However, you might want to test with Flume 1.3.0 which was just released. We actually use the CDH version of Flume from Cloudera and so our tests had many of the fixes that are in 1.3. I'm not sure what if what we were measuring is what you are. We tested the performance difference is what the application sees, not the time to make it to the repository.
In our tests the remote agent is running on a remote machine where the latency is most likely the primary reason the embedded appender performs so much better.
Also, since you are using a vanilla pattern layout your log events may be smaller than ours. We are using StructuredDataMessages and each of the items in the Message are added as a header to the FlumeEvent, in addition to being included in the message. This allows them to be efficiently inserted into Cassandra.
Ralph
On Dec 4, 2012, at 3:44 AM, SimonMo wrote:
> Hi Ralph,
>
> can u give me your properties for this test.
> i've got the same speed for both appenders.
> and the second problem with the embedded: if i send 20.000 logs i only get
> ~17.000 logs.
>
> embedded:
> <Flume name="embedded" suppressExceptions="false" compress="false"
> embedded="true" batchsize="1">
> <Property name="channels">file</Property>
> <Property name="channels.file.type">file</Property>
> <Property
> name="channels.file.checkpointDir">filechannel/checkpoint</Property>
> <Property name="channels.file.dataDirs">filechannel/data</Property>
> <Property name="sinks">agent1</Property>
> <Property name="sinks.agent1.channel">file</Property>
> <Property name="sinks.agent1.type">avro</Property>
> <Property name="sinks.agent1.hostname">127.0.0.1</Property>
> <Property name="sinks.agent1.port">41414</Property>
> <PatternLayout pattern="%m" />
> </Flume>
>
> not embedded:
> <Flume name="flumeAppender" suppressExceptions="false" compress="false"
> embedded="false">
> <Agent host="127.0.0.1" port="42424" />
> <PatternLayout>
> <pattern>%m</pattern>
> </PatternLayout>
> </Flume>
>
> Regards,
> Simon
>
>
>
>
>
> --
> View this message in context: http://apache-logging.6191.n7.nabble.com/Flume-Appender-tp7089p35073.html
> Sent from the Log4j - Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Flume Appender
Posted by SimonMo <si...@gmail.com>.
Hi Ralph,
can u give me your properties for this test.
i've got the same speed for both appenders.
and the second problem with the embedded: if i send 20.000 logs i only get
~17.000 logs.
embedded:
<Flume name="embedded" suppressExceptions="false" compress="false"
embedded="true" batchsize="1">
<Property name="channels">file</Property>
<Property name="channels.file.type">file</Property>
<Property
name="channels.file.checkpointDir">filechannel/checkpoint</Property>
<Property name="channels.file.dataDirs">filechannel/data</Property>
<Property name="sinks">agent1</Property>
<Property name="sinks.agent1.channel">file</Property>
<Property name="sinks.agent1.type">avro</Property>
<Property name="sinks.agent1.hostname">127.0.0.1</Property>
<Property name="sinks.agent1.port">41414</Property>
<PatternLayout pattern="%m" />
</Flume>
not embedded:
<Flume name="flumeAppender" suppressExceptions="false" compress="false"
embedded="false">
<Agent host="127.0.0.1" port="42424" />
<PatternLayout>
<pattern>%m</pattern>
</PatternLayout>
</Flume>
Regards,
Simon
--
View this message in context: http://apache-logging.6191.n7.nabble.com/Flume-Appender-tp7089p35073.html
Sent from the Log4j - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Flume Appender
Posted by Ralph Goers <ra...@dslextreme.com>.
As Christian pointed out, Log4j 2 is still considered beta quality software. While it works fine in the environments I have tested in others are starting to use it in other environments and as they encounter problems and fixes are implemented the quality will improve.
That said, I am using the Flume Appender and we recently did some performance testing. The chart below shows the result of a single application writing the specified number of events. The application and remote agents are running in VMs, which is not going to yield the best performance Flume has to offer. However, our application doesn't really reach a very large scale until we start aggregating events from a number of applications into Flume Agents acting as collectors. We don't expect a single application to be writing more than about 10 events per second so the numbers below are just fine even in the non-embedded case. The agent collectors can be run on bare metal and achieve a fairly high level of performance. In addition, you can run a cluster of agents in your collector pool and point the applications to different agents to scale up to the numbers you are talking about. Of course, your ultimate destination also has to support that kind of rate. I am not familiar with Elasticsearch. For more discussion on how to get Flume to scale I would suggest asking on the Flume mailing lists.
As you can see, the embedded agent is significantly faster than the remote agent. The downside though, is that when you want to update Flume your application has to be modified.
I had a discussion with the Flume developers just last week about the embedded appender. The current release of Flume doesn't support embedded out of the box so I had to do a bit of work. In one of the next releases they plan on improving the support. That really should impact the visible portion of the Appender but should make it easier to implement and maintain.
HTH
Ralph
On Oct 25, 2012, at 6:34 AM, SimonMo wrote:
>
> Hi,
>
> does anybody know how stable is the flume appender embedded?
> I have a network with nodes logging 20.000 logs/sec and i want to use Log4j
> 2 -> Flume -> Elasticsearch.
> Is this a good approach?
>
> I could use additionally a FailoverAppender to get my logs on a second route
> to my server...
> hmm, i'm not sure.
>
> Has anybody experience with so much logs in an network?
>
> Regards,
> Simon
> --
> View this message in context: http://old.nabble.com/Flume-Appender-tp34601088p34601088.html
> Sent from the Log4j - Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>