You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/10/22 05:25:54 UTC
[PR] HDDS-16822. Write buffered stream data to the block file before committing a PutBlock [ozone]
peterxcli opened a new pull request, #11678:
URL: https://github.com/apache/ozone/pull/11678
## What changes were proposed in this pull request?
With Ratis streaming, the datanode keeps the last 1 MiB or more of each stream in memory (the `Buffers` of `KeyValueStreamDataChannel`). It needs these bytes at close, because the end of the stream may be the PutBlock request that the client appends. By default (`ozone.client.datastream.putblock.without.raft.enabled=false`), `hsync()` commits its PutBlock through Raft, and nothing writes the buffered bytes to the block file first. HDDS-16746 fixed this only for a PutBlock sent as a stream command.
So when `hsync()` returns, every replica has committed a block length whose last 1 MiB or more is not in its block file. For example, after writing 3 MiB + 1000 bytes and calling `hsync()`, each block file has 2 MiB:
- A reader that opens the file fails with `Unexpected read size. expected: 262144, actual: -1`.
- If the writer fails before it closes the stream, the bytes never reach the block file. Lease recovery then closes the file at the hsync'ed length, and the file stays unreadable.
With this change:
- `KeyValueHandler.handlePutBlock` asks the chunk manager to write the buffered data of the block's open stream, up to the length that the PutBlock commits, before it commits the PutBlock.
- `FilePerBlockStrategy` keeps the open stream channels by container and local ID. A channel removes itself when it is closed or cleaned up.
- `KeyValueStreamDataChannel.drainBuffers(long)` writes whole buffers until the block file has the given length, and stops there. At close the client sends the PutBlock request at the end of the stream and also a PutBlock through Raft, so writing all the buffers could put that request into the block file. The client sends the request in its own packets after the data, so writing whole buffers up to the committed length is safe.
- The Raft PutBlock is applied on a different thread than the stream writes, so the methods that use the buffers are synchronized. The PutBlock at the end of the stream is committed without holding the lock.
The drain is a buffered write without fsync, which is what an ordinary hsync gets with the default `hdds.container.chunk.write.sync=false`.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16822
## How was this patch tested?
- `TestContainerStateMachineStream#testReadAfterHsync` is the existing `testReadAfterHsyncWithPutBlockWithoutRaft`, now run in both PutBlock modes and with 1000 bytes and 3 MiB + 1000 bytes. It writes, calls `hsync()` and reads the key from a second client. The Raft cases failed before this change.
- New `TestLeaseRecovery#testRecoveryAfterStreamWriterFailure` calls `hsync()`, closes the writer's connection to the datanodes without closing the stream, recovers the lease from a second client and reads the file. The Raft case failed before this change.
- New unit tests `TestKeyValueStreamDataChannel#testDrainBuffersUpToLength`, `#testDrainBuffersAfterCleanUp` and `TestFilePerBlockStrategy#testDrainStreamBuffers`.
- Locally, these also pass: `TestBlockDataStreamOutput` (integration and unit), `TestHSync`, `TestOzoneFileSystemWithStreaming`, and the container-service unit tests for the stream channel, the chunk managers, `KeyValueHandler` and `ContainerStateMachine`. Checkstyle is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16822. Write buffered stream data to the block file before committing a PutBlock [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #11678:
URL: https://github.com/apache/ozone/pull/11678#discussion_r4345232710
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestLeaseRecovery.java:
##########
@@ -226,6 +232,38 @@ public void testRecovery(int dataSize) throws Exception {
verifyData(data, dataSize * 2, file, fs);
}
+ @ParameterizedTest
+ @ValueSource(booleans = {false, true})
Review Comment:
nit: the `true` case was already fixed by HDDS-16746, and `TestContainerStateMachineStream#testReadAfterHsync` covers both modes. Since this test also depends on several client internals (`SelectorOutputStream`, `OzoneFSDataStreamOutput`, `KeyDataStreamOutput`), consider running only the Raft PutBlock mode here.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -675,6 +675,7 @@ ContainerCommandResponseProto handlePutBlock(
ContainerProtos.BlockData data = request.getPutBlock().getBlockData();
BlockData blockData = BlockData.getFromProtoBuf(data);
Objects.requireNonNull(blockData, "blockData == null");
+ chunkManager.drainStreamBuffers(kvContainer, blockData.getBlockID(), blockData.getSize());
Review Comment:
nit: a comment here would help readers who do not know about the stream buffering, e.g.
```java
// A data stream keeps the end of the data in memory; it must be in the block file before the length is committed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16822. Write buffered stream data to the block file before committing a PutBlock [ozone]
Posted by "Copilot (via GitHub) via issues" <is...@ozone.apache.org>.
Copilot commented on code in PR #11678:
URL: https://github.com/apache/ozone/pull/11678#discussion_r4560266453
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java:
##########
@@ -171,6 +178,34 @@ public void drainBuffers() throws IOException {
}
}
+ /**
+ * Write the data in {@link #buffers} to the channel until the block file has at least the given length.
+ * It must be called before committing a PutBlock with this block length.
+ * Unlike {@link #drainBuffers()}, it does not write all the data:
+ * the end of the stream may be the PutBlock request, which must not be written to the block file.
+ * Since the client sends the PutBlock request in separate packets after the data,
+ * whole buffers can be written as long as the block file is shorter than the given length.
+ */
+ synchronized void drainBuffers(long length) throws IOException {
+ if (closed.get()) {
+ // close() has written the data, or cleanUp() has dropped it.
+ return;
Review Comment:
`close()` sets `closed` before acquiring the buffer monitor, so this check can skip draining while the tail is still only in memory. Since the client submits the Raft PutBlock without waiting for stream close, that interleaving can commit a block length ahead of its data. Make the close-state transition and tail write atomic with draining, keep the PutBlock callback outside the monitor, and add a deterministic close-versus-drain regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16822. Write buffered stream data to the block file before committing a PutBlock [ozone]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@ozone.apache.org>.
github-actions[bot] commented on PR #11678:
URL: https://github.com/apache/ozone/pull/11678#issuecomment-6839685781
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org