You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "amaliujia (via GitHub)" <gi...@apache.org> on 2026/10/08 14:34:42 UTC
[PR] DDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
amaliujia opened a new pull request, #11446:
URL: https://github.com/apache/ozone/pull/11446
## What changes were proposed in this pull request?
`Buffers.pollAll()` in the KeyValue stream write path throws `IllegalStateException` when the internal deque is empty (`Preconditions.checkState(!deque.isEmpty(), ...)`).
Callers such as `KeyValueStreamDataChannel.drainBuffers()` must guard with `buffers.isEmpty()` before calling `pollAll()`, which duplicates checks and makes the API easy to misuse.
This PR proposes when the deque is empty, `pollAll()` returns `ReferenceCountedObject.wrap(Unpooled.EMPTY_BUFFER)` instead of throwing.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16789
## How was this patch tested?
Unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #11446:
URL: https://github.com/apache/ozone/pull/11446#discussion_r4220758402
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java:
##########
@@ -144,9 +144,11 @@ private void writeBuffers() throws IOException {
final ReferenceCountedObject<ByteBuf> ref = buffers.pollAll();
final ByteBuf buf = ref.retain();
try {
- setEndIndex(buf);
- // write the remaining data
- writeFully(buf.nioBuffer(), super::writeFileChannel);
+ if (buf.isReadable()) {
Review Comment:
<img width="735" height="91" alt="Screenshot 2026-10-08 at 23 12 49" src="https://github.com/user-attachments/assets/3f43480e-a4c4-427a-80db-3bc06c71d7ec" />
The empty ByteBuf is neither writable nor readable so we cannot apply `setEndIndex` over it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() allows empty buffer instead of failing on empty deque [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #11446:
URL: https://github.com/apache/ozone/pull/11446#issuecomment-6199907219
> ... do you have any idea that if we can refactor setEndIndex so we dont need additional flag for pollAll
Sorry, I don't get the question. Having a isEmpty flag seems good -- for write, we expect that it must have some data to write; for drain, it could be empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #11446:
URL: https://github.com/apache/ozone/pull/11446#issuecomment-6155338674
>
I might be wrong for this suggestion, but looks like we dont have a clean way to really get rid of the empty check(isReadable or isEmpty). and main problem is setEndIndex need at least four bytes to know how many data to read
@szetszwo do you have any idea that if we can refactor `setEndIndex` so we dont need additional flag for `pollAll`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() allows empty buffer instead of failing on empty deque [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia merged PR #11446:
URL: https://github.com/apache/ozone/pull/11446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on PR #11446:
URL: https://github.com/apache/ozone/pull/11446#issuecomment-6138398606
@peterxcli @szetszwo thanks for the review.
I applied the suggestion that we can allow `drainBuffers` work over empty buffers while write/close will still hit the exception as what it is now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #11446:
URL: https://github.com/apache/ozone/pull/11446#discussion_r4251100310
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/Buffers.java:
##########
@@ -90,7 +90,9 @@ ReferenceCountedObject<ByteBuffer> poll() {
}
ReferenceCountedObject<ByteBuf> pollAll() {
- Preconditions.checkState(!deque.isEmpty(), "The deque is empty");
+ if (deque.isEmpty()) {
+ return ReferenceCountedObject.wrap(Unpooled.EMPTY_BUFFER);
Review Comment:
@amaliujia , thanks for working on this! Some suggestions:
- Let's add a boolean allowEmpty parameter. Set it to false for the write case and true for the drainBuffers.
- When it is empty, just return null and check null in drainBuffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #11446:
URL: https://github.com/apache/ozone/pull/11446#issuecomment-6079139243
I might wrong for this suggestion, but looks like we dont have a clean way to really get rid of the empty check(`isReadable` or `isEmpty`). and main problem is `setEndIndex` need at least four bytes to know how many data to read
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #11446:
URL: https://github.com/apache/ozone/pull/11446#discussion_r4220758402
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java:
##########
@@ -144,9 +144,11 @@ private void writeBuffers() throws IOException {
final ReferenceCountedObject<ByteBuf> ref = buffers.pollAll();
final ByteBuf buf = ref.retain();
try {
- setEndIndex(buf);
- // write the remaining data
- writeFully(buf.nioBuffer(), super::writeFileChannel);
+ if (buf.isReadable()) {
Review Comment:
<img width="735" height="91" alt="Screenshot 2026-10-08 at 23 12 49" src="https://github.com/user-attachments/assets/3f43480e-a4c4-427a-80db-3bc06c71d7ec" />
The empty buf is neither writable nor readable so we cannot apply `setEndIndex` over it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #11446:
URL: https://github.com/apache/ozone/pull/11446#discussion_r4220758402
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java:
##########
@@ -144,9 +144,11 @@ private void writeBuffers() throws IOException {
final ReferenceCountedObject<ByteBuf> ref = buffers.pollAll();
final ByteBuf buf = ref.retain();
try {
- setEndIndex(buf);
- // write the remaining data
- writeFully(buf.nioBuffer(), super::writeFileChannel);
+ if (buf.isReadable()) {
Review Comment:
<img width="735" height="91" alt="Screenshot 2026-10-08 at 23 12 49" src="https://github.com/user-attachments/assets/3f43480e-a4c4-427a-80db-3bc06c71d7ec" />
The empty buffer is neither writable or readable so we cannot apply `setEndIndex` over it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() allows empty buffer instead of failing on empty deque [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on PR #11446:
URL: https://github.com/apache/ozone/pull/11446#issuecomment-6182699663
Merging this PR now. We could always follow-up if there is new ideas for improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #11446:
URL: https://github.com/apache/ozone/pull/11446#discussion_r4220693052
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java:
##########
@@ -144,9 +144,11 @@ private void writeBuffers() throws IOException {
final ReferenceCountedObject<ByteBuf> ref = buffers.pollAll();
final ByteBuf buf = ref.retain();
try {
- setEndIndex(buf);
- // write the remaining data
- writeFully(buf.nioBuffer(), super::writeFileChannel);
+ if (buf.isReadable()) {
Review Comment:
curious why adding this check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on PR #11446:
URL: https://github.com/apache/ozone/pull/11446#issuecomment-6104581265
@peterxcli will it still be a benefit if getting ride of an unnecessary exception and allow the code continue working when facing empty buffer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16789. Make Buffers.pollAll() return empty buffer instead of failing on empty deque [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #11446:
URL: https://github.com/apache/ozone/pull/11446#discussion_r4251100310
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/Buffers.java:
##########
@@ -90,7 +90,9 @@ ReferenceCountedObject<ByteBuffer> poll() {
}
ReferenceCountedObject<ByteBuf> pollAll() {
- Preconditions.checkState(!deque.isEmpty(), "The deque is empty");
+ if (deque.isEmpty()) {
+ return ReferenceCountedObject.wrap(Unpooled.EMPTY_BUFFER);
Review Comment:
@amaliujia , thanks for working on this! Some suggestions:
- Let's add a boolean allowEmpty parameter. Set it to false for the write case and true for the drainBuffers.
- When it is empty, just return null and check null in drainBuffers.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/Buffers.java:
##########
@@ -90,7 +90,9 @@ ReferenceCountedObject<ByteBuffer> poll() {
}
ReferenceCountedObject<ByteBuf> pollAll() {
- Preconditions.checkState(!deque.isEmpty(), "The deque is empty");
+ if (deque.isEmpty()) {
+ return ReferenceCountedObject.wrap(Unpooled.EMPTY_BUFFER);
Review Comment:
@amaliujia , thanks for working on this! Some suggestions:
- Let's add a boolean allowEmpty parameter. Set it to false for the write case and true for the drainBuffers.
- When it is empty, just return null and check null in drainBuffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org