You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-user@logging.apache.org by Paul Duffy <pa...@cisco.com> on 2007/08/02 01:23:52 UTC
Looking for high speed binary appender?
Folks,
My team is looking at log4j as a next generation logging option, but we
have a specific customer need to do high volume logging which a text
based system may not support. Is anyone aware of a high speed binary
logging capability that builds upon the log4j infrastructure (or otherwise?
Cheers
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Looking for high speed binary appender?
Posted by James Stauffer <st...@gmail.com>.
What is your destination? Files, database, JMS queue? Do you have
any idea how much logging you will be doing?
On 8/1/07, Paul Duffy <pa...@cisco.com> wrote:
> Folks,
>
> My team is looking at log4j as a next generation logging option, but we
> have a specific customer need to do high volume logging which a text
> based system may not support. Is anyone aware of a high speed binary
> logging capability that builds upon the log4j infrastructure (or otherwise?
>
> Cheers
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
--
James Stauffer http://www.geocities.com/stauffer_james/
Are you good? Take the test at http://www.livingwaters.com/good/
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
RE: [BULK] Re: Looking for high speed binary appender?
Posted by Philip Denno <pd...@tsi.bc.ca>.
My current project may be of interest to you.
I have a Java App which runs about 600 threads logging to files using
the RollingFileAppender.
Now this app under heavy load writes out about 600 MB of logging every 4
hours when configured at DEBUG. This app is deployed on a four - core
dual processor running Linux. Under these conditions the app peaks at a
CPU usage of about 10%.
When I change the level to WARN I hardly see any change in the CPU
profile, so logging forms a very small percentage of the load.
What is the projected amount of logging that you will be doing. Unless
you are writing GB's every hour I think log4j should be able to handle
it.
Cheers,
Philip.
-----Original Message-----
From: Paul Duffy [mailto:paduffy@cisco.com]
Sent: Thursday, August 02, 2007 11:02 AM
To: Log4J Users List
Subject: [BULK] Re: Looking for high speed binary appender?
Importance: Low
Folks,
To clarify...
- my team's current app (highly distributed, born in 98) is using legacy
home grown text based logging,
- we're looking at an upgrade to log4j.
- but major customers are saying they want to increase log column, from
(essentially moving from WARN to INFO level).
I do not have specific volume info, only that my gut tells he we'll be
heading into territory that log4j may not be able to handle.
There is talk within my team of going to a more binary based logging to
cut down text manipulation costs, disk space, etc. In essence, defer
message formatting overhead from runtime to post processing. Obviously
we prefer to use any robust open source solutions available.
Then there is this from the NetLogger folks...
http://dsd.lbl.gov/publications/HPDC02-HP-monitoring.pdf
...which describes a binary based logging framework which seriously
outperforms log4j.
So my questions are
- has anyone built a high volume (binary or otherwise) optimized logging
facility atop log4j?
- what other solutions have been applied when log4j performance is an
issue?
Cheers
Paul Smith wrote:
>
> On 02/08/2007, at 9:23 AM, Paul Duffy wrote:
>
>> Folks,
>>
>> My team is looking at log4j as a next generation logging option, but
>> we have a specific customer need to do high volume logging which a
>> text based system may not support. Is anyone aware of a high speed
>> binary logging capability that builds upon the log4j infrastructure
>> (or otherwise?
>>
>
> You'd be surprised how many events a file-based appender can write.
> Can you provide some #'s on how many events you think you'll need to
> support?
>
> Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
__________________
IMPORTANT NOTICE
Email from TSI Terminal Systems Inc. (TSI) is confidential and may be legally privileged. If it is not intended for you, please delete it immediately unread. Under no circumstances should this e-mail be redistributed without prior written consent of TSI. The internet cannot guarantee that this communication is free of viruses, interception or interference and anyone who communicates with us by email is taken to accept the risks in doing so. Without limitation, TSI and its affiliates accept no liability whatsoever and howsoever arising in connection with the use of this email. Under no circumstances shall this email constitute a binding agreement for provision of services by TSI, which is subject to the terms and conditions of TSI's standard schedule for Terminal Tariffs which is also available at http://www.tsi.bc.ca.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Looking for high speed binary appender?
Posted by Paul Duffy <pa...@cisco.com>.
Folks,
To clarify...
- my team's current app (highly distributed, born in 98) is using legacy
home grown text based logging,
- we're looking at an upgrade to log4j.
- but major customers are saying they want to increase log column, from
(essentially moving from WARN to INFO level).
I do not have specific volume info, only that my gut tells he we'll be
heading into territory that log4j may not be able to handle.
There is talk within my team of going to a more binary based logging to
cut down text manipulation costs, disk space, etc. In essence, defer
message formatting overhead from runtime to post processing. Obviously
we prefer to use any robust open source solutions available.
Then there is this from the NetLogger folks...
http://dsd.lbl.gov/publications/HPDC02-HP-monitoring.pdf
...which describes a binary based logging framework which seriously
outperforms log4j.
So my questions are
- has anyone built a high volume (binary or otherwise) optimized logging
facility atop log4j?
- what other solutions have been applied when log4j performance is an issue?
Cheers
Paul Smith wrote:
>
> On 02/08/2007, at 9:23 AM, Paul Duffy wrote:
>
>> Folks,
>>
>> My team is looking at log4j as a next generation logging option, but
>> we have a specific customer need to do high volume logging which a
>> text based system may not support. Is anyone aware of a high speed
>> binary logging capability that builds upon the log4j infrastructure
>> (or otherwise?
>>
>
> You'd be surprised how many events a file-based appender can write.
> Can you provide some #'s on how many events you think you'll need to
> support?
>
> Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Looking for high speed binary appender?
Posted by Paul Smith <ps...@aconex.com>.
On 02/08/2007, at 9:23 AM, Paul Duffy wrote:
> Folks,
>
> My team is looking at log4j as a next generation logging option,
> but we have a specific customer need to do high volume logging
> which a text based system may not support. Is anyone aware of a
> high speed binary logging capability that builds upon the log4j
> infrastructure (or otherwise?
>
You'd be surprised how many events a file-based appender can write.
Can you provide some #'s on how many events you think you'll need to
support?
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org