You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "yandrey321 (via GitHub)" <gi...@apache.org> on 2026/06/04 23:22:17 UTC
[PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
yandrey321 opened a new pull request, #10439:
URL: https://github.com/apache/ozone/pull/10439
Remove excessive debug logging in BlockOutputStream and BufferPool
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15486
## How was this patch tested?
Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3360284831
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -94,14 +85,12 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
releaser.start();
allocator.join();
assertEquals(toRelease, allocated.get());
- assertTrue(logCapturer.getOutput().contains("Allocation needs to wait the pool is at capacity"));
Review Comment:
I think we are losing testing coverage if you remove this line.
We might can do a
```
assertTrue(pool.allocated.size() == pool.capacity)
allocated.set(pool.allocateBuffer(0));
```
[above](https://github.com/apache/ozone/pull/10439/changes#diff-e42410e41f0e7d2b824f6ad6c1461d9f85da655e0b862bf149df60d0d8ccc488R66-R67)
However I am not completely sure.
To be safe, we can keep this type of verification. Other removals over LOG_DEBUG look ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4930776851
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3366352027
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -94,14 +85,12 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
releaser.start();
allocator.join();
assertEquals(toRelease, allocated.get());
- assertTrue(logCapturer.getOutput().contains("Allocation needs to wait the pool is at capacity"));
Review Comment:
Good point. Yes `assertFull(pool)` does state checks.
I am wondering if other people can also take a look at this before we merge. cc @adoroszlai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4675125877
@yandrey321 , Since this is BlockOutputStream, the performance benchmark should include network IO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4652179096
> For the hot path the best practice is to limit amount of logging unless its placed for catching specific issue. JIT can do a lot of optimization and simplification/reducing amount of ifs on the hot path helps with JIT's execution path analysis as well as with low level optimizations.
If this is a performance improvement, please provide some numbers and describe how to run the benchmarks.
In Ozone, we use LOG.debug(..) everywhere. Why removing only from BlockOutputStream but not the other classes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4658812966
> If this is a performance improvement, please provide some numbers and describe how to run the benchmarks.
Also, please try the same benchmark after setting `log4j.threshold=INFO` in `log4j.properties`, but without any code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4683830519
Sorry that I am -1 on this change.
- Again, the debug messages are very useful for debugging
- There is no performance improvement in this change due to network IO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3363716254
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -94,14 +85,12 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
releaser.start();
allocator.join();
assertEquals(toRelease, allocated.get());
- assertTrue(logCapturer.getOutput().contains("Allocation needs to wait the pool is at capacity"));
Review Comment:
there is a check if pool is full at the beginning of the function:
`assertFull(pool);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3360284831
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -94,14 +85,12 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
releaser.start();
allocator.join();
assertEquals(toRelease, allocated.get());
- assertTrue(logCapturer.getOutput().contains("Allocation needs to wait the pool is at capacity"));
Review Comment:
I think we are losing testing coverage if you remove this line.
We might be able to do the following [above](https://github.com/apache/ozone/pull/10439/changes#diff-e42410e41f0e7d2b824f6ad6c1461d9f85da655e0b862bf149df60d0d8ccc488R66-R67)
```
assertTrue(pool.allocated.size() == pool.capacity)
allocated.set(pool.allocateBuffer(0));
```
However I am not completely sure.
To be safe, we can keep this type of verification. Other removals over LOG_DEBUG look ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3360284831
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -94,14 +85,12 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
releaser.start();
allocator.join();
assertEquals(toRelease, allocated.get());
- assertTrue(logCapturer.getOutput().contains("Allocation needs to wait the pool is at capacity"));
Review Comment:
I think we are losing testing coverage if you remove this line.
We might be able to do
```
assertTrue(pool.allocated.size() == pool.capacity)
allocated.set(pool.allocateBuffer(0));
```
[above](https://github.com/apache/ozone/pull/10439/changes#diff-e42410e41f0e7d2b824f6ad6c1461d9f85da655e0b862bf149df60d0d8ccc488R66-R67)
However I am not completely sure.
To be safe, we can keep this type of verification. Other removals over LOG_DEBUG look ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4662760048
> > For the hot path the best practice is to limit amount of logging unless its placed for catching specific issue. JIT can do a lot of optimization and simplification/reducing amount of ifs on the hot path helps with JIT's execution path analysis as well as with low level optimizations.
>
> If this is a performance improvement, please provide some numbers and describe how to run the benchmarks.
>
> In Ozone, we use LOG.debug(..) everywhere. Why removing only from BlockOutputStream but not the other classes?
in general LOG.debug() is OK when they are not on the hot path and do not do expensive calls for logging params. This PR is a follow up after profiling freon benchmark and removes debug logs inside frames that consumes 5+% of CPU during the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10439: HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu…
URL: https://github.com/apache/ozone/pull/10439
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3363487379
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -94,14 +85,12 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
releaser.start();
allocator.join();
assertEquals(toRelease, allocated.get());
- assertTrue(logCapturer.getOutput().contains("Allocation needs to wait the pool is at capacity"));
Review Comment:
I'd avoid validation through log messages and prefer to do it via actual state check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4650518120
> @yandrey321 , Why removing the debug messages? They can be enabled/disabled by setting the log level of BlockOutputStream.
>
> When there is a bugs, debug messages are very helpful.
For the hot path the best practice is to limit amount of logging unless its placed for catching specific issue. JIT can do a lot of optimization and simplification/reducing amount of ifs on the hot path helps with JIT's execution path analysis as well as with low level optimizations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4680776443
> @yandrey321 , Since this is BlockOutputStream, the performance benchmark should include network IO.
Its not an end-to-end benchmark for BlockOutputStream, the goal of the benchmark was to isolate the client side of the BlockOutputStream and show that removing debug logging on the hot path slightly affects the throughput if backend side is a constant (which is never true in the real environment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3366352027
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -94,14 +85,12 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
releaser.start();
allocator.join();
assertEquals(toRelease, allocated.get());
- assertTrue(logCapturer.getOutput().contains("Allocation needs to wait the pool is at capacity"));
Review Comment:
Good point. Yes `assertFull(pool)` does state checks.
I am wondering if other people can also take a look at this before we merge. cc @adoroszlai.
This is just in case if there was hidden context for such log based testing that we do not know. And certainly if bugs exist in such code path (e.g. a while loop + wait in BufferPool), it is relatively hard to debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3366352027
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -94,14 +85,12 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
releaser.start();
allocator.join();
assertEquals(toRelease, allocated.get());
- assertTrue(logCapturer.getOutput().contains("Allocation needs to wait the pool is at capacity"));
Review Comment:
Good point. Yes `assertFull(pool)` does state checks.
I am wondering if other people can also take a look at this before we merge. cc @adoroszlai.
This is just in case if there was hidden context for such log based testing that we do not know. And certainly if bugs exist in such code path (e.g. a while loop + wait in PoolBuffer), it is relatively hard to debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3366355780
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -59,6 +59,8 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
// As the pool is full, allocation will need to wait until a buffer is released.
assertFull(pool);
+ assertEquals(buffers.size(), pool.getAllocatedBuffers().size());
Review Comment:
Looks like this addition verifies if there is any new allocation happened after a batch of buffer allocations through the pool.
Can we name this assertion as`assertNoBuffersChangeInPool(buffersSize, pool)`? This might have better readability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10439:
URL: https://github.com/apache/ozone/pull/10439#discussion_r3366355780
##########
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java:
##########
@@ -59,6 +59,8 @@ private void assertAllocationBlockedUntilReleased(BufferPool pool, Deque<ChunkBu
// As the pool is full, allocation will need to wait until a buffer is released.
assertFull(pool);
+ assertEquals(buffers.size(), pool.getAllocatedBuffers().size());
Review Comment:
Can we consolidate this line into `assertFull`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4662774372
> > If this is a performance improvement, please provide some numbers and describe how to run the benchmarks.
>
> Also, please try the same benchmark after setting `log4j.threshold=INFO` in `log4j.properties`, but without any code change.
I added benchmark for writing to the output stream and test results before and after the changes (run on mac with ARM/m3 CPU)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4671863728
> benchmark shows 4% throughput improvement
I'm not sure about that, or that a single data point for each scenario is enough to conclude anything.
I ran the benchmark with 3 scenarios:
- 1. `master` before this change
- 2. with this change
- 3. `master` before this change with `log4j.threshold=INFO`
each 3 times.
Results:
```
1-master/1.log:throughput: 5,629.4 MB/s
1-master/2.log:throughput: 5,997.3 MB/s
1-master/3.log:throughput: 5,570.6 MB/s
2-fix/1.log:throughput: 5,565.2 MB/s
2-fix/2.log:throughput: 5,528.3 MB/s
2-fix/3.log:throughput: 5,710.9 MB/s
3-threshold/1.log:throughput: 5,551.3 MB/s
3-threshold/2.log:throughput: 5,596.5 MB/s
3-threshold/3.log:throughput: 5,667.1 MB/s
```
Unexpectedly `master` has the highest mean throughput, but also the highest variance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15486 Remove excessive debug logging in BlockOutputStream and Bu… [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on PR #10439:
URL: https://github.com/apache/ozone/pull/10439#issuecomment-4671963040
> 5,667.1
I'm not sure why master shows better numbers, it might have some other changes that affect the numbers.
For posted numbers I run the benchmark with the changes, then reverted back the changes in the workspace and run again. Benchmark was run multiple times and it was consistently showed better numbers with removed debug logging.
Please note that the benchmark results can be affected by the background processes running on the system. The interesting thing here, for my mac the numbers were 17-18k MB/s, in your tests 5-6k range, for a some reason they are 3x lower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org