You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ptlrs (via GitHub)" <gi...@apache.org> on 2025/11/07 21:55:56 UTC
[PR] HDDS-13894. Add benchmark results for async and sync audit loggers. [ozone]
ptlrs opened a new pull request, #9264:
URL: https://github.com/apache/ozone/pull/9264
## What changes were proposed in this pull request?
Benchmark results for async and sync audit loggers required for https://github.com/apache/ozone/pull/9229
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13894
## How was this patch tested?
No tests needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13894. Add benchmark results for async and sync audit loggers. [ozone]
Posted by "ptlrs (via GitHub)" <gi...@apache.org>.
ptlrs closed pull request #9264: HDDS-13894. Add benchmark results for async and sync audit loggers.
URL: https://github.com/apache/ozone/pull/9264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13894. Add benchmark results for async and sync audit loggers. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #9264:
URL: https://github.com/apache/ozone/pull/9264#issuecomment-3509401097
Thanks @ptlrs for the benchmark.
> However, I don't think this belongs to design docs.
Agreed, I think we can put the benchmark results under HDDS-13794 Jira ticket instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13894. Add benchmark results for async and sync audit loggers. [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #9264:
URL: https://github.com/apache/ozone/pull/9264#issuecomment-3513092657
@yandrey321 do you wanna review this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13894. Add benchmark results for async and sync audit loggers. [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #9264:
URL: https://github.com/apache/ozone/pull/9264#discussion_r2511438044
##########
hadoop-hdds/docs/content/design/logger-benchmark.md:
##########
@@ -0,0 +1,64 @@
+---
+title: Audit Logger Benchmarks
+summary: Benchmarking the async and sync audit loggers using log4j2
+date: 2025-11-02
+jira: HDDS-13894
+status: implemented
+author: Rishabh Patel
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+## Introduction & Goal
+
+This document presents the performance benchmark results comparing our current Asynchronous (Async) Logger with
+a new Synchronous (Sync) Logger configuration in Log4j2.
+The primary goal of this testing is to validate the performance impact of transitioning our audit logging framework
+to a Sync Logger.
+This transition is critical to guarantee the integrity and completeness of our audit logs, as the current Async Logger
+can drop log events under high stress.
+
+
+## Benchmark Methodology
+To evaluate the impact, we ran a series of tests using the ozone freon tool to simulate the OM being under pressure.
+
+- We used a bare-metal cluster with a few configuration tweaks to perform these tests.
+
+- The OM log directory was moved from an SSD to an HDD to expose any issues when the disk is slow.
Review Comment:
what is our typical destination for audit logs? Do we put logs to the system drive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13894. Add benchmark results for async and sync audit loggers. [ozone]
Posted by "ptlrs (via GitHub)" <gi...@apache.org>.
ptlrs commented on code in PR #9264:
URL: https://github.com/apache/ozone/pull/9264#discussion_r2586258974
##########
hadoop-hdds/docs/content/design/logger-benchmark.md:
##########
@@ -0,0 +1,64 @@
+---
+title: Audit Logger Benchmarks
+summary: Benchmarking the async and sync audit loggers using log4j2
+date: 2025-11-02
+jira: HDDS-13894
+status: implemented
+author: Rishabh Patel
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+## Introduction & Goal
+
+This document presents the performance benchmark results comparing our current Asynchronous (Async) Logger with
+a new Synchronous (Sync) Logger configuration in Log4j2.
+The primary goal of this testing is to validate the performance impact of transitioning our audit logging framework
+to a Sync Logger.
+This transition is critical to guarantee the integrity and completeness of our audit logs, as the current Async Logger
+can drop log events under high stress.
+
+
+## Benchmark Methodology
+To evaluate the impact, we ran a series of tests using the ozone freon tool to simulate the OM being under pressure.
+
+- We used a bare-metal cluster with a few configuration tweaks to perform these tests.
+
+- The OM log directory was moved from an SSD to an HDD to expose any issues when the disk is slow.
Review Comment:
Yes, the logs go to the system drives currently which are supposed to be SSDs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13894. Add benchmark results for async and sync audit loggers. [ozone]
Posted by "ptlrs (via GitHub)" <gi...@apache.org>.
ptlrs commented on PR #9264:
URL: https://github.com/apache/ozone/pull/9264#issuecomment-3608371871
Thanks for the review @adoroszlai @ivandika3 @yandrey321
The benchmark markdown file has been uploaded to both https://issues.apache.org/jira/browse/HDDS-13794 and https://issues.apache.org/jira/browse/HDDS-13894.
Closing this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13894. Add benchmark results for async and sync audit loggers. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9264:
URL: https://github.com/apache/ozone/pull/9264#issuecomment-3599515425
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org