You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-user@logging.apache.org by Andreas Guther <an...@guther.com> on 2004/06/18 14:27:54 UTC
Re: [log4j] FW: Database Logging Performance Numbers
I am also interested in that kind of information. I also started recently
looking into logging into tables and would be interested in some performance
results and experience reports in high volume environments.
Andreas
pooja wrote:
>Hi All
>
>I haven't received a response on this issue.
>Can someone at Log4j answer it for me?
>
>Thanx
>--- Pooja.
>
>-----Original Message-----
>From: pooja [mailto:pooja@induslogic.com]
>Sent: Monday, June 14, 2004 6:50 PM
>To: 'Log4J Users List'
>Subject: Database Logging Performance Numbers
>
>Hello
>
>I am looking for some performance numbers for Database Logging.
>
>I want to be able to analyze/query my log records at a later date in
>multiple of ways. Hence Logging into database seems to be the perfect
>choice. But the system is a transactional system and it will be handling
>more than a million transactions a day. Hence making it necessary, that
>I first prove that Logging into Database will not hamper performance.
>Can someone provide some published numbers?
>
>I will be fine with test cases as well; I'll run them against the
>current file logging system and get comparative figures. Currently, I am
>writing some crude test cases of my own.
>
>At the same time, if neone knows of some application (which handles so
>many transactions) already writing logs into database & functioning,
>this reference will suffice my job.
>
>Thanx
>--- Pooja.
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: [log4j] FW: Database Logging Performance Numbers
Posted by Andreas Guther <an...@guther.com>.
Ceki:
thanks for your answer. Sounds very interesting to me. Until you have
time to finish that project and to publish the results I probably have
to see if I can write my own enhanced version in the meantime. But
same problem here: time and priorities.
Andreas
Ceki Gülcü wrote:
> Andreas,
>
> Since you asked, I am talking about a specialized version of
> DBAppender which I have *not* published because it uses advanced JDBC
> features unavailable on most JDBC drivers. Actually, it is supposed to
> work only with the most recent version of the MySQL JDBC connector.
> (On older JDBC connectors there were problems.) The idea is to buffer
> the events and spit them out in batches. This results in a huge boost
> in performance. However, I currently do not have the time to
> consolidate the code as there are much more urgent items on my to do
> list.
>
> At 04:21 AM 6/23/2004, Andreas Guther wrote:
>
>> Ceki:
>>
>> Is this a specific (MySQL) DBAppender you are talking about?
>>
>> Andreas
>>
>> Ceki Gülcü wrote:
>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: [log4j] FW: Database Logging Performance Numbers
Posted by Ceki Gülcü <ce...@qos.ch>.
Andreas,
Since you asked, I am talking about a specialized version of DBAppender
which I have *not* published because it uses advanced JDBC features
unavailable on most JDBC drivers. Actually, it is supposed to work only
with the most recent version of the MySQL JDBC connector. (On older JDBC
connectors there were problems.) The idea is to buffer the events and spit
them out in batches. This results in a huge boost in performance. However,
I currently do not have the time to consolidate the code as there are much
more urgent items on my to do list.
At 04:21 AM 6/23/2004, Andreas Guther wrote:
>Ceki:
>
>Is this a specific (MySQL) DBAppender you are talking about?
>
>Andreas
>
>Ceki Gülcü wrote:
>
>>
>>My tests with DBAppender show that using pooled connections, it takes
>>about 10 milliseconds to insert an event into a MySQL DB (of type
>>*INNODB*) running locally. It still takes approx. 10 millis logging to a
>>PostgreSQL DB running on a relatively old and slow Linux server (with
>>pooled connections).
>>
>>DBAppender is completely DB agnostic. It will run on all RDBMMSs. An
>>optimized version of DBAppender (but which is MySQL-only) logs at 500
>>*microseconds* per event.
>>
>>HTH,
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>For additional commands, e-mail: log4j-user-help@logging.apache.org
--
Ceki Gülcü
For log4j documentation consider "The complete log4j manual"
ISBN: 2970036908 http://www.qos.ch/shop/products/clm_t.jsp
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: [log4j] FW: Database Logging Performance Numbers
Posted by Andreas Guther <an...@guther.com>.
Ceki:
Is this a specific (MySQL) DBAppender you are talking about?
Andreas
Ceki Gülcü wrote:
>
> My tests with DBAppender show that using pooled connections, it takes
> about 10 milliseconds to insert an event into a MySQL DB (of type
> *INNODB*) running locally. It still takes approx. 10 millis logging to
> a PostgreSQL DB running on a relatively old and slow Linux server
> (with pooled connections).
>
> DBAppender is completely DB agnostic. It will run on all RDBMMSs. An
> optimized version of DBAppender (but which is MySQL-only) logs at 500
> *microseconds* per event.
>
> HTH,
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: [log4j] FW: Database Logging Performance Numbers
Posted by Ceki Gülcü <ce...@qos.ch>.
My tests with DBAppender show that using pooled connections, it takes about
10 milliseconds to insert an event into a MySQL DB (of type *INNODB*)
running locally. It still takes approx. 10 millis logging to a PostgreSQL
DB running on a relatively old and slow Linux server (with pooled connections).
DBAppender is completely DB agnostic. It will run on all RDBMMSs. An
optimized version of DBAppender (but which is MySQL-only) logs at 500
*microseconds* per event.
HTH,
At 02:27 PM 6/18/2004, you wrote:
>I am also interested in that kind of information. I also started recently
>looking into logging into tables and would be interested in some performance
>results and experience reports in high volume environments.
>
>Andreas
>
>
>pooja wrote:
>
>>Hi All
>>I haven't received a response on this issue. Can someone at Log4j answer
>>it for me?
>>Thanx
>>--- Pooja.
>>-----Original Message-----
>>From: pooja [mailto:pooja@induslogic.com] Sent: Monday, June 14, 2004 6:50 PM
>>To: 'Log4J Users List'
>>Subject: Database Logging Performance Numbers
>>
>>Hello
>>I am looking for some performance numbers for Database Logging.
>>I want to be able to analyze/query my log records at a later date in
>>multiple of ways. Hence Logging into database seems to be the perfect
>>choice. But the system is a transactional system and it will be handling
>>more than a million transactions a day. Hence making it necessary, that
>>I first prove that Logging into Database will not hamper performance.
>>Can someone provide some published numbers?
>>
>>I will be fine with test cases as well; I'll run them against the
>>current file logging system and get comparative figures. Currently, I am
>>writing some crude test cases of my own.
>>At the same time, if neone knows of some application (which handles so
>>many transactions) already writing logs into database & functioning,
>>this reference will suffice my job.
>>Thanx
>>--- Pooja.
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>For additional commands, e-mail: log4j-user-help@logging.apache.org
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>For additional commands, e-mail: log4j-user-help@logging.apache.org
>>
>>
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>For additional commands, e-mail: log4j-user-help@logging.apache.org
--
Ceki Gülcü
For log4j documentation consider "The complete log4j manual"
ISBN: 2970036908 http://www.qos.ch/shop/products/clm_t.jsp
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: [log4j] FW: Database Logging Performance Numbers
Posted by Thomas Svensen <ts...@cellus.no>.
Since noone else seems to have any information on this, I can tell you
what I heard from a guy working for telco company with a rather huge
number of transactions.
They did _not_ get acceptable performance and ended up logging
insert-statements via FileAppender and PatternLayout, and then
batch-inserting these on regular intervals.
A more manual solution, of course, but a guarantee that db performance
will not influence critical production performance. Anyway, I guess db
performance will rely mostly on non-log4j related parameters like
networking, db configuration etc.
Just wanted to let you know :)
- Thomas S.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org