You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-user@logging.apache.org by Paul Smith <ps...@aconex.com> on 2008/09/04 03:31:23 UTC

network logging performance

If anyone out there is using "Logging over the network" of any form  
(socket, JMS, Multicast, syslog appenders etc), this topic is for you.  
I'm wondering whether people could comment on their experience setting  
up high performance logging of application data over the network.  The  
cost of shipping an event over the wire compared with writing it to a  
local file is obviously higher, and so one can notice user-visible  
impact if the logging volume is high when using network-based logging  
(unless one wraps it in AsyncAppenders).

Just curious to hear peoples experiences, strategies, and thoughts.   
Perhaps people can relate to the flow rate they've manage to achieve  
under different configurations.

The driver to my question is that for my Apache Lab project (Pinpoint)  
I'm building a central logging repository server to allow data mining  
of the generated logging events.  In a high performance site one can  
have many logging events shipped from multiple threads in, say, a web- 
app, being shipped serially over the wire, so slowing down the logging  
can slow down the response time.

cheers,

Paul Smith

---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org


RE: network logging performance

Posted by Bruno Melloni <Br...@wnco.com>.
**Theoretically**, a logging event should be a 'fire and forget'
activity from the perspective of the application.  That means that you
should be able to submit the logging event to the 'logging system'
without blocking, and then the 'logging system' should be able to
deliver those logging events to the final destination on its own
thread(s) and at its own speed.  As long as the average volume of
logging does not exceed the capacity of the actual logging mechanism
(i.e.: over the network) then there ought to be no impact whatsoever to
the application (except perhaps for the cpu overhead of the physical
logging thread(s)).

I don't know log4j well enough to say whether it is already doing this
isolation of 'log event submission' vs 'log event processing'.  If it
doesn't, you should be able to have your application submit all logging
requests through a class written by you that puts all requests into a
memory cache, ensuring no blocking.  Then have one or more separate
threads handling the 'log event processing' independently from the rest
of the app.  For safety, you should probably add some code that monitors
memory, disk and network utilization by this approach and notify you
(perhaps by email) when things get backed up - that way you'll detect
any volume-caused issues before they have a chance to impact the app.

If you really wanted to be bulletproof, implement the cache to use
memory first (up to some limit) and then overflow to a local file cache.
That way if your network gets bogged down by issues other than your
app's you would still be able to continue functioning while the network
recovers.

I know that did not answer your immediate question, but it should help
some in making your choices.

bruno

-----Original Message-----
From: Paul Smith [mailto:psmith@aconex.com] 
Sent: Wednesday, September 03, 2008 8:31 PM
To: Log4J Users List
Subject: network logging performance

If anyone out there is using "Logging over the network" of any form
(socket, JMS, Multicast, syslog appenders etc), this topic is for you.  
I'm wondering whether people could comment on their experience setting
up high performance logging of application data over the network.  The
cost of shipping an event over the wire compared with writing it to a
local file is obviously higher, and so one can notice user-visible
impact if the logging volume is high when using network-based logging
(unless one wraps it in AsyncAppenders).

Just curious to hear peoples experiences, strategies, and thoughts.   
Perhaps people can relate to the flow rate they've manage to achieve
under different configurations.

The driver to my question is that for my Apache Lab project (Pinpoint)
I'm building a central logging repository server to allow data mining of
the generated logging events.  In a high performance site one can have
many logging events shipped from multiple threads in, say, a web- app,
being shipped serially over the wire, so slowing down the logging can
slow down the response time.

cheers,

Paul Smith

---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org


RE: network logging performance

Posted by Michael Erskine <mi...@ketech.com>.
> If anyone out there is using "Logging over the network" of any form
> (socket, JMS, Multicast, syslog appenders etc), this topic is for you.
> I'm wondering whether people could comment on their experience setting
> up high performance logging of application data over the network.

For convenience I routinely configure SocketHubAppender instances for all my services and typically set a threshold of WARN to reduce traffic to important info.

Where performance is deemed more important than the timely delivery of the information I have custom asynchronous appenders with limited BlockingDeque queues and daemon threads to pump out data at their leisure.

Where I have multiple services logging across the network to a central place I tend to use a custom JDBC asynchronous appender.

In other scenarios I have networked systems that require certain log messages as part of a larger communications protocol. Here I have custom appenders that catch LoggingEvent messages, translate them and drop them in the outgoing BlockingDeque for the target system (I always work with queues on complex systems to encourage loose coupling).

I don't tend to pass Java objects over the wire: call me a traditionalist but when I write my protocols they tend to be human-readable 7-bit ASCII where you can telnet to a server, issue commands and get replies. This is great for unit testing, troubleshooting, etc. More recently I've been writing systems that interoperate with 3rd party systems where XML is passed over the wire (so performance is not of major importance :) ) and here I tend to make Apache XMLBeans and StAX parsers do all the work but internally I'll drop down to a human readable (and loggable!) protocol for testing and faster interfacing.

I have a strong belief that any non-trivial networked communications system can be generalised to IRC!


Just my 2 euros :)

Regards,
Michael Erskine.


---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org


Re: network logging performance

Posted by Paul Smith <ps...@aconex.com>.
[oops, I originally sent this reply direct to Johannes.  Copying again  
the list]

On 04/09/2008, at 4:19 PM, Johannes Gutleber wrote:

> Dear Paul,
>
> On Sep 4, 2008, at 03:31 , Paul Smith wrote:
>
>> If anyone out there is using "Logging over the network" of any form  
>> (socket, JMS, Multicast, syslog appenders etc), this topic is for  
>> you. I'm wondering whether people could comment on their experience  
>> setting up high performance logging of application data over the  
>> network.  The cost of shipping an event over the wire compared with  
>> writing it to a local file is obviously higher, and so one can  
>> notice user-visible impact if the logging volume is high when using  
>> network-based logging (unless one wraps it in AsyncAppenders).
>
> In fact at CERN we're working with a 2000 computer, 10'000  
> applications systems now for about 2 years in which we deploy also  
> logging over the network using a
> combination of log4cplus and log4j using the XML data format.
>

fantastic.  awesome to hear about this sort of environment.  Really  
appreciate you taking the time to respond in depth here.

>
> The experience with the tools at hand is rather disappointing in  
> that the problem is not the log emitter, but the task of log  
> collection. For that purpose we use a JAVA/Tomcat based log  
> collector that
> has the capability to forward the collected logs to multiple  
> outputs: SocketHub appender, OracleDB, JMS, etc. The collector was  
> supposed to implement filtering and prioritizing capabilities, but
> those have practically all been dropped since they just added to the  
> performance bottleneck instead of avoiding it (we experienced that  
> Java is simply not performant enough to do this job
> for a size of system that we have). People went on about a  
> hierarchical layout of log collectors, but that would not resolve  
> the bottlenecks.
>

Are you able to articulate where you thought the bottleneck within the  
Java process was ?  What JRE were you using?  Was it GC overhead?  Was  
it log4j itself, or the custom built collector that was the problem?


> To that log collector we attach JMS subscribers as well as Chainsaw  
> log viewers for online viewing.
>
> Both the log-collector and the Chainsaw claints fail frequently due  
> to performance problems of Java and extensive memory usage.
>

Which version of Chainsaw did you use? Was it the one inbuilt into  
log4j, or the newer Chainsaw v2?  The latter uses a cyclic buffer so  
as not to consume too much memory.  I do agree though that even  
Chainsaw v2 is far from perfect.. :)

>
> So finally after about 2.5 years we stepped back to write to local  
> files and collect/inspect them when needed. We replaced on-line  
> logging with an error propagation system
> based on an XML protocol and WS-Eventing publisher subscriber. All  
> implemented in C++, no Java anywhere. A testing phase  of 2 month  
> has proven successful and we
> deploy the system since 1 month without major problems.
>
> Of course my mail reflects the situation of a very peculiar, large  
> size distributed soft-real time system that is probably not so  
> common in other fields.
> In addition we wanted to use on-line log viewing and analysis, which  
> is also not the common case in other domains.


My vision, ridiculous as this may sound, is that log4j should be able  
to support an environment in the new cloud environments.  Imagine the  
Hadoop clusters that are out there and making sense of wtf happened  
during the dissemination of the MapReduce portions, with thousands of  
host nodes logging and, somehow, an engineer able to 'see' what  
happened on his job centrally (even if it's not real time).

With a Many->1 collector the collector host is going to have be one  
grunty box....

Pinpoints current design uses ActiveMQ internally.  The Receivers  
inside accept the remote event and place it on the local log4j bus  
(just like it was placed on the remote bus that went to the network  
appender).  Internally a local  'appender' routes the event to an  
internal JMS topic, with ActiveMQ buffering the events to a local  
persistent store temporarily.  A single topic listener then tries to  
chew through the received events to index them (indexing is always  
going to be expensive hence a producer/consumer pattern).  This  
temporary JMS buffer I am hoping will allow the receiving of events  
much faster than they can be consumed which is sort of what JMS (and  
ActiveMQ) is designed to handle, although it'll be interesting to see  
how it translate into practice.  Where I work that is driving me to  
develop Pinpoint has no where near the load environment that yours has  
though.

I'd love to hear some actual logging numbers, that is, how many events  
(log lines would do) that each host on average is producing and their  
combined total (even a peak load for a given hour or something would  
be interesting).  Your environment really is exactly what I had in  
mind to support with Pinpoint, and I'm very much focussed on making it  
as non-intrusive as possible.

cheers,

Paul Smith


---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org