You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@logging.apache.org by GitBox <gi...@apache.org> on 2021/12/22 03:14:15 UTC

[GitHub] [logging-log4cxx] rm5248 opened a new pull request #87: Imported and slightly cleaned up throughput tests

rm5248 opened a new pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87


   This PR adds in a utility used for benchmarking the log4cxx library.  It does not benchmark anything about actually writing data out; that is outside of the scope of what this benchmarks at the moment(as that can be highly variable depending on the OS).
   
   This is made as a separate utility application, as it is not really a unit test and is more of a test of how well the library itself performs.
   
   Sample output:
   ```
   Benchmarking library only(no writing out):
   **************************************************************
   Benchmarking Single threaded: 1000000 messages
   **************************************************************
   Log4cxx NoFormat pattern: %m%n Elapsed: 0.9181 secs 1,089,168/sec
   Log4cxx DateOnly pattern: [%d] %m%n Elapsed: 0.8941 secs 1,118,448/sec
   Log4cxx DateClassLevel pattern: [%d] [%c] [%p] %m%n Elapsed: 0.8908 secs 1,122,597/sec
   Log4cxx Logging with FMT Elapsed: 0.4779 secs 2,092,464/sec
   Log4cxx Logging static string Elapsed: 0.4279 secs 2,337,184/sec
   Log4cxx Logging static string with FMT Elapsed: 0.4834 secs 2,068,577/sec
   Log4cxx Logging disabled debug Elapsed: 0.04615 secs 21,670,312/sec
   Log4cxx Logging disabled trace Elapsed: 0.04780 secs 20,921,870/sec
   Log4cxx Logging enabled debug Elapsed: 0.4301 secs 2,325,013/sec
   Log4cxx Logging enabled trace Elapsed: 0.4288 secs 2,332,359/sec
   **************************************************************
   Benchmarking multithreaded threaded: 1000000 messages/thread, 4 threads
   **************************************************************
   Log4cxx Logging with FMT MT Elapsed: 2.869 secs 348,524/sec
   Log4cxx Logging with FMT MT Elapsed: 2.937 secs 340,445/sec
   Log4cxx Logging with FMT MT Elapsed: 2.956 secs 338,264/sec
   Log4cxx Logging with FMT MT Elapsed: 2.968 secs 336,947/sec
   **************************************************************
   Benchmarking multithreaded disabled: 1000000 messages/thread, 4 threads
   **************************************************************
   Log4cxx Logging disabled MT Elapsed: 0.3615 secs 2,766,610/sec
   Log4cxx Logging disabled MT Elapsed: 0.3884 secs 2,574,516/sec
   Log4cxx Logging disabled MT Elapsed: 0.4346 secs 2,300,853/sec
   Log4cxx Logging disabled MT Elapsed: 0.4362 secs 2,292,636/sec
   Results for use in spreadsheet:
   1089168
   1118448
   1122597
   2092464
   2337184
   2068577
   21670312
   20921870
   2325013
   2332359
   348524
   340445
   338264
   336947
   2766610
   2574516
   2300853
   2292636
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] swebb2066 commented on pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
swebb2066 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000584520


   > Part of the performance testing is to also help us find areas where we can better parallelize the logging. There are probably a number of places we can replace a mutex with a shared_mutex.
   
   Multiple threaded applications are serialised by Logger::isDisabled because Logger now uses a weak pointer (commit 565fdea38) to the LoggerRepository. This is a significant regression with respect to version 11 that we need to think about carefully. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] swebb2066 commented on pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
swebb2066 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000882788


   Your data shows the rate of disabled log calls for multithread case which should be near the single thread rate for each thread.
   
   On Dec 25, 2021 1:38 AM, Robert Middleton ***@***.***> wrote:
   
   I'll have to take a look at that; I would be surprised if that was a significant source of performance issues, but some testing would confirm that.
   
   FWIW, using the LOG4CXX_[level]_FMT family of macros and also the fact that we use a thread_local for the thread name<https://github.com/apache/logging-log4cxx/commit/aaaea6aadf3fda0d2e0511d7034305cf1e8b9353> should more than compensate for the extra time locking the weak_ptr.
   
   —
   Reply to this email directly, view it on GitHub<https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000862347>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADJDRC77R6BQ5WWZJFNMLXTUSSAT3ANCNFSM5KRSIELQ>.
   Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
   You are receiving this because you commented.Message ID: ***@***.***>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] rm5248 commented on pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
rm5248 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000862347


   I'll have to take a look at that; I would be surprised if that was a significant source of performance issues, but some testing would confirm that.
   
   FWIW, using the `LOG4CXX_[level]_FMT` family of macros and also the fact that we use a `thread_local` for the [thread name](https://github.com/apache/logging-log4cxx/commit/aaaea6aadf3fda0d2e0511d7034305cf1e8b9353) should more than compensate for the extra time locking the `weak_ptr`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] swebb2066 edited a comment on pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
swebb2066 edited a comment on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-999933135


   Hi Robert,
   
   This should do the initial job just fine. 
   
   However, a couple of things occur to me in hindsight.
   
   1. Multi threaded logging to a single logger probably should be part of the unit tests rather than a throughput test. The mutex in log4cxx::Hierarchy (or AppenderAttachableImpl::appendLoopOnAppenders) effectively serialize the threads, so the results are not interesting. Also, in multi-thread applications, each thread is likely to use a different log4cxx::logger instance. I see the MultithreadTest unit test you added in March 2021 covers the single logger case.
   
   2. It would be nice to be able to easily monitor changes in throughput in the future as changes (PRs) are proposed. I see in https://medium.com/nerd-for-tech/ci-build-performance-testing-with-github-action-e6b227097c83 there is already support available. Would you like me to spend some time looking at the options? How do you feel about introducing another dependency for CI benchmarking?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] rm5248 commented on pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
rm5248 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000014049


   >     1. Multi threaded logging to a single logger probably should be part of the unit tests rather than a throughput test. The mutex in log4cxx::Hierarchy (or AppenderAttachableImpl::appendLoopOnAppenders) effectively serialize the threads, so the results are not interesting. Also, in multi-thread applications, each thread is likely to use a different log4cxx::logger instance. I see the MultithreadTest unit test you added in March 2021 covers the single logger case.
   > 
   
   That is a good thought, I'll look into adding that test in the next few days.  I think it really depends on how you're logging as well, if you have a logger instance per-class or a logger instance per-thread(these are not mutually exclusive).  I do agree that there definitely needs to be more multithreaded unit tests to help identify issues with threading.
   
   Part of the performance testing is to also help us find areas where we can better parallelize the logging.  There are probably a number of places we can replace a `mutex` with a `shared_mutex`.
   
   >     2. It would be nice to be able to easily monitor changes in throughput in the future as changes (PRs) are proposed. I see in https://medium.com/nerd-for-tech/ci-build-performance-testing-with-github-action-e6b227097c83 there is already support available. Would you like me to spend some time looking at the options? How do you feel about introducing another dependency for CI benchmarking?
   
   If you could do that, that would be great!  Note that within the ASF repos, we can't use any actions except for the offical github actions, so it may need to be something custom.  At least saving the data from the run would at least be a step in the right direction.  As long as any options in CMake are configurable by the user so they don't need to run them by default, that would be great.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] swebb2066 commented on a change in pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
swebb2066 commented on a change in pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#discussion_r774235555



##########
File path: src/test/cpp/throughput/CMakeLists.txt
##########
@@ -0,0 +1,40 @@
+option(BUILD_THROUGHPUT_TESTS "Build throughput tests"  OFF)
+
+if( NOT BUILD_THROUGHPUT_TESTS )

Review comment:
       Also exit with a message if the FMT library was found




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] swebb2066 commented on pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
swebb2066 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-999933135


   Hi Robert,
   
   This should do the initial job just fine. 
   
   However, a couple of things occur to me in hindsight.
   
   1. Multi threaded logging to a single logger probably should be part of the unit tests rather than a throughput test. The mutex in log4cxx::Hierarchy effectively serialize the threads, so the results are not interesting. Also, in multi-thread applications, each thread is likely to use a different log4cxx::logger instance. I see the MultithreadTest unit test you added in March 2021 covers the single logger case.
   
   2. It would be nice to be able to easily monitor changes in throughput in the future as changes (PRs) are proposed. I see in https://medium.com/nerd-for-tech/ci-build-performance-testing-with-github-action-e6b227097c83 there is already support available. Would you like me to spend some time looking at the options? How do you feel about introducing another dependency for CI benchmarking?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] swebb2066 edited a comment on pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
swebb2066 edited a comment on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000882788


   Your data shows the rate of disabled log calls for multithread case which should be near the single thread rate for each thread.
   
   I checked version 11 and the rate of disabled log calls for multithread case is near the single thread rate for each thread.
   
   I created an issue in Jira for this (LOG4CXX-546)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] rm5248 merged pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
rm5248 merged pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] rm5248 commented on a change in pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
rm5248 commented on a change in pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#discussion_r774285680



##########
File path: src/test/cpp/throughput/CMakeLists.txt
##########
@@ -0,0 +1,40 @@
+option(BUILD_THROUGHPUT_TESTS "Build throughput tests"  OFF)
+
+if( NOT BUILD_THROUGHPUT_TESTS )

Review comment:
       Since it's marked as required, it will do that already.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [logging-log4cxx] rm5248 edited a comment on pull request #87: Imported and slightly cleaned up throughput tests

Posted by GitBox <gi...@apache.org>.
rm5248 edited a comment on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000014049


   > 1.  Multi threaded logging to a single logger probably should be part of the unit tests rather than a throughput test. The mutex in log4cxx::Hierarchy (or AppenderAttachableImpl::appendLoopOnAppenders) effectively serialize the threads, so the results are not interesting. Also, in multi-thread applications, each thread is likely to use a different log4cxx::logger instance. I see the MultithreadTest unit test you added in March 2021 covers the single logger case.
   > 
   
   That is a good thought, I'll look into adding that test in the next few days.  I think it really depends on how you're logging as well, if you have a logger instance per-class or a logger instance per-thread(these are not mutually exclusive).  I do agree that there definitely needs to be more multithreaded unit tests to help identify issues with threading.
   
   Part of the performance testing is to also help us find areas where we can better parallelize the logging.  There are probably a number of places we can replace a `mutex` with a `shared_mutex`.
   
   >  2. It would be nice to be able to easily monitor changes in throughput in the future as changes (PRs) are proposed. I see in https://medium.com/nerd-for-tech/ci-build-performance-testing-with-github-action-e6b227097c83 there is already support available. Would you like me to spend some time looking at the options? How do you feel about introducing another dependency for CI benchmarking?
   
   If you could do that, that would be great!  Note that within the ASF repos, we can't use any actions except for the offical github actions, so it may need to be something custom.  At least saving the data from the run would at least be a step in the right direction.  As long as any options in CMake are configurable by the user so they don't need to run them by default, that would be great.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org