You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@logging.apache.org by GitBox <gi...@apache.org> on 2021/12/22 03:14:15 UTC
[GitHub] [logging-log4cxx] rm5248 opened a new pull request #87: Imported and slightly cleaned up throughput tests
rm5248 opened a new pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87
This PR adds in a utility used for benchmarking the log4cxx library. It does not benchmark anything about actually writing data out; that is outside of the scope of what this benchmarks at the moment(as that can be highly variable depending on the OS).
This is made as a separate utility application, as it is not really a unit test and is more of a test of how well the library itself performs.
Sample output:
```
Benchmarking library only(no writing out):
**************************************************************
Benchmarking Single threaded: 1000000 messages
**************************************************************
Log4cxx NoFormat pattern: %m%n Elapsed: 0.9181 secs 1,089,168/sec
Log4cxx DateOnly pattern: [%d] %m%n Elapsed: 0.8941 secs 1,118,448/sec
Log4cxx DateClassLevel pattern: [%d] [%c] [%p] %m%n Elapsed: 0.8908 secs 1,122,597/sec
Log4cxx Logging with FMT Elapsed: 0.4779 secs 2,092,464/sec
Log4cxx Logging static string Elapsed: 0.4279 secs 2,337,184/sec
Log4cxx Logging static string with FMT Elapsed: 0.4834 secs 2,068,577/sec
Log4cxx Logging disabled debug Elapsed: 0.04615 secs 21,670,312/sec
Log4cxx Logging disabled trace Elapsed: 0.04780 secs 20,921,870/sec
Log4cxx Logging enabled debug Elapsed: 0.4301 secs 2,325,013/sec
Log4cxx Logging enabled trace Elapsed: 0.4288 secs 2,332,359/sec
**************************************************************
Benchmarking multithreaded threaded: 1000000 messages/thread, 4 threads
**************************************************************
Log4cxx Logging with FMT MT Elapsed: 2.869 secs 348,524/sec
Log4cxx Logging with FMT MT Elapsed: 2.937 secs 340,445/sec
Log4cxx Logging with FMT MT Elapsed: 2.956 secs 338,264/sec
Log4cxx Logging with FMT MT Elapsed: 2.968 secs 336,947/sec
**************************************************************
Benchmarking multithreaded disabled: 1000000 messages/thread, 4 threads
**************************************************************
Log4cxx Logging disabled MT Elapsed: 0.3615 secs 2,766,610/sec
Log4cxx Logging disabled MT Elapsed: 0.3884 secs 2,574,516/sec
Log4cxx Logging disabled MT Elapsed: 0.4346 secs 2,300,853/sec
Log4cxx Logging disabled MT Elapsed: 0.4362 secs 2,292,636/sec
Results for use in spreadsheet:
1089168
1118448
1122597
2092464
2337184
2068577
21670312
20921870
2325013
2332359
348524
340445
338264
336947
2766610
2574516
2300853
2292636
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] swebb2066 commented on pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
swebb2066 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000584520
> Part of the performance testing is to also help us find areas where we can better parallelize the logging. There are probably a number of places we can replace a mutex with a shared_mutex.
Multiple threaded applications are serialised by Logger::isDisabled because Logger now uses a weak pointer (commit 565fdea38) to the LoggerRepository. This is a significant regression with respect to version 11 that we need to think about carefully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] swebb2066 commented on pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
swebb2066 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000882788
Your data shows the rate of disabled log calls for multithread case which should be near the single thread rate for each thread.
On Dec 25, 2021 1:38 AM, Robert Middleton ***@***.***> wrote:
I'll have to take a look at that; I would be surprised if that was a significant source of performance issues, but some testing would confirm that.
FWIW, using the LOG4CXX_[level]_FMT family of macros and also the fact that we use a thread_local for the thread name<https://github.com/apache/logging-log4cxx/commit/aaaea6aadf3fda0d2e0511d7034305cf1e8b9353> should more than compensate for the extra time locking the weak_ptr.
—
Reply to this email directly, view it on GitHub<https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000862347>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADJDRC77R6BQ5WWZJFNMLXTUSSAT3ANCNFSM5KRSIELQ>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you commented.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] rm5248 commented on pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
rm5248 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000862347
I'll have to take a look at that; I would be surprised if that was a significant source of performance issues, but some testing would confirm that.
FWIW, using the `LOG4CXX_[level]_FMT` family of macros and also the fact that we use a `thread_local` for the [thread name](https://github.com/apache/logging-log4cxx/commit/aaaea6aadf3fda0d2e0511d7034305cf1e8b9353) should more than compensate for the extra time locking the `weak_ptr`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] swebb2066 edited a comment on pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
swebb2066 edited a comment on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-999933135
Hi Robert,
This should do the initial job just fine.
However, a couple of things occur to me in hindsight.
1. Multi threaded logging to a single logger probably should be part of the unit tests rather than a throughput test. The mutex in log4cxx::Hierarchy (or AppenderAttachableImpl::appendLoopOnAppenders) effectively serialize the threads, so the results are not interesting. Also, in multi-thread applications, each thread is likely to use a different log4cxx::logger instance. I see the MultithreadTest unit test you added in March 2021 covers the single logger case.
2. It would be nice to be able to easily monitor changes in throughput in the future as changes (PRs) are proposed. I see in https://medium.com/nerd-for-tech/ci-build-performance-testing-with-github-action-e6b227097c83 there is already support available. Would you like me to spend some time looking at the options? How do you feel about introducing another dependency for CI benchmarking?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] rm5248 commented on pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
rm5248 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000014049
> 1. Multi threaded logging to a single logger probably should be part of the unit tests rather than a throughput test. The mutex in log4cxx::Hierarchy (or AppenderAttachableImpl::appendLoopOnAppenders) effectively serialize the threads, so the results are not interesting. Also, in multi-thread applications, each thread is likely to use a different log4cxx::logger instance. I see the MultithreadTest unit test you added in March 2021 covers the single logger case.
>
That is a good thought, I'll look into adding that test in the next few days. I think it really depends on how you're logging as well, if you have a logger instance per-class or a logger instance per-thread(these are not mutually exclusive). I do agree that there definitely needs to be more multithreaded unit tests to help identify issues with threading.
Part of the performance testing is to also help us find areas where we can better parallelize the logging. There are probably a number of places we can replace a `mutex` with a `shared_mutex`.
> 2. It would be nice to be able to easily monitor changes in throughput in the future as changes (PRs) are proposed. I see in https://medium.com/nerd-for-tech/ci-build-performance-testing-with-github-action-e6b227097c83 there is already support available. Would you like me to spend some time looking at the options? How do you feel about introducing another dependency for CI benchmarking?
If you could do that, that would be great! Note that within the ASF repos, we can't use any actions except for the offical github actions, so it may need to be something custom. At least saving the data from the run would at least be a step in the right direction. As long as any options in CMake are configurable by the user so they don't need to run them by default, that would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] swebb2066 commented on a change in pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
swebb2066 commented on a change in pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#discussion_r774235555
##########
File path: src/test/cpp/throughput/CMakeLists.txt
##########
@@ -0,0 +1,40 @@
+option(BUILD_THROUGHPUT_TESTS "Build throughput tests" OFF)
+
+if( NOT BUILD_THROUGHPUT_TESTS )
Review comment:
Also exit with a message if the FMT library was found
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] swebb2066 commented on pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
swebb2066 commented on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-999933135
Hi Robert,
This should do the initial job just fine.
However, a couple of things occur to me in hindsight.
1. Multi threaded logging to a single logger probably should be part of the unit tests rather than a throughput test. The mutex in log4cxx::Hierarchy effectively serialize the threads, so the results are not interesting. Also, in multi-thread applications, each thread is likely to use a different log4cxx::logger instance. I see the MultithreadTest unit test you added in March 2021 covers the single logger case.
2. It would be nice to be able to easily monitor changes in throughput in the future as changes (PRs) are proposed. I see in https://medium.com/nerd-for-tech/ci-build-performance-testing-with-github-action-e6b227097c83 there is already support available. Would you like me to spend some time looking at the options? How do you feel about introducing another dependency for CI benchmarking?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] swebb2066 edited a comment on pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
swebb2066 edited a comment on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000882788
Your data shows the rate of disabled log calls for multithread case which should be near the single thread rate for each thread.
I checked version 11 and the rate of disabled log calls for multithread case is near the single thread rate for each thread.
I created an issue in Jira for this (LOG4CXX-546)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] rm5248 merged pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
rm5248 merged pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] rm5248 commented on a change in pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
rm5248 commented on a change in pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#discussion_r774285680
##########
File path: src/test/cpp/throughput/CMakeLists.txt
##########
@@ -0,0 +1,40 @@
+option(BUILD_THROUGHPUT_TESTS "Build throughput tests" OFF)
+
+if( NOT BUILD_THROUGHPUT_TESTS )
Review comment:
Since it's marked as required, it will do that already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4cxx] rm5248 edited a comment on pull request #87: Imported and slightly cleaned up throughput tests
Posted by GitBox <gi...@apache.org>.
rm5248 edited a comment on pull request #87:
URL: https://github.com/apache/logging-log4cxx/pull/87#issuecomment-1000014049
> 1. Multi threaded logging to a single logger probably should be part of the unit tests rather than a throughput test. The mutex in log4cxx::Hierarchy (or AppenderAttachableImpl::appendLoopOnAppenders) effectively serialize the threads, so the results are not interesting. Also, in multi-thread applications, each thread is likely to use a different log4cxx::logger instance. I see the MultithreadTest unit test you added in March 2021 covers the single logger case.
>
That is a good thought, I'll look into adding that test in the next few days. I think it really depends on how you're logging as well, if you have a logger instance per-class or a logger instance per-thread(these are not mutually exclusive). I do agree that there definitely needs to be more multithreaded unit tests to help identify issues with threading.
Part of the performance testing is to also help us find areas where we can better parallelize the logging. There are probably a number of places we can replace a `mutex` with a `shared_mutex`.
> 2. It would be nice to be able to easily monitor changes in throughput in the future as changes (PRs) are proposed. I see in https://medium.com/nerd-for-tech/ci-build-performance-testing-with-github-action-e6b227097c83 there is already support available. Would you like me to spend some time looking at the options? How do you feel about introducing another dependency for CI benchmarking?
If you could do that, that would be great! Note that within the ASF repos, we can't use any actions except for the offical github actions, so it may need to be something custom. At least saving the data from the run would at least be a step in the right direction. As long as any options in CMake are configurable by the user so they don't need to run them by default, that would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org