You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "henrybear327 (via GitHub)" <gi...@apache.org> on 2026/07/21 23:22:59 UTC
[PR] HDDS-15930. Avoid zero-length chunk in TestContainerOperations [ozone]
henrybear327 opened a new pull request, #10833:
URL: https://github.com/apache/ozone/pull/10833
## What changes were proposed in this pull request?
This PR fixes the [flake](https://github.com/henrybear327/ozone/actions/runs/29770924105/job/88455733500), which is the `TestContainerOperations#testContainerStateMachineIdempotency`, by ensuring that its randomly generated chunk is non-empty.
#### Root cause investigation
Taken from the CI run log.
The failing CI run showed that the test generated a zero-byte chunk:
```text
2026-07-20 21:42:37,572 [ForkJoinPool-1-worker-1] DEBUG scm.XceiverClientGrpc (XceiverClientGrpc.java:sendCommandWithRetry(495)) - Executing command cmdType: WriteChunk traceID: "" containerID: 67 datanodeUuid: "8a3fc17b-fb86-4d79-9be3-3d51133d4f50" writeChunk { blockID { containerID: 67 localID: 116954481136238947 blockCommitSequenceId: 0 } chunkData { chunkName: "116954481136238947.data.0" offset: 0 len: 0 } data: "<redacted>" } version: 3 on datanode 8a3fc17b-fb86-4d79-9be3-3d51133d4f50(localhost/127.0.0.1)
```
The following `PutBlock` also described an empty block:
```text
2026-07-20 21:42:37,578 [ForkJoinPool-1-worker-1] DEBUG scm.XceiverClientGrpc (XceiverClientGrpc.java:sendCommandWithRetry(495)) - Executing command cmdType: PutBlock traceID: "" containerID: 67 datanodeUuid: "8a3fc17b-fb86-4d79-9be3-3d51133d4f50" putBlock { blockData { blockID { containerID: 67 localID: 116954481136238947 blockCommitSequenceId: 0 } chunks { chunkName: "116954481136238947.data.0" offset: 0 len: 0 } size: 0 } } version: 3 on datanode 8a3fc17b-fb86-4d79-9be3-3d51133d4f50(localhost/127.0.0.1)
```
This led to the following failure sequence:
```text
2026-07-20 21:42:37,580 [8a3fc17b-fb86-4d79-9be3-3d51133d4f50-ChunkReader-2] WARN keyvalue.KeyValueHandler (ContainerUtils.java:logAndReturnError(99)) - Operation: PutBlock , Trace ID: , Message: Chunk file not found: /home/runner/work/ozone/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-3f677bbb-a268-48d5-a8e5-e030d249b108/ozone-metadata/datanode-5/data-0/hdds/3f677bbb-a268-48d5-a8e5-e030d249b108/current/containerDir0/67/chunks/116954481136238947.block , Result: UNABLE_TO_FIND_CHUNK , StorageContainerException Occurred.
```
then
```text
Marked container UNHEALTHY from OPEN
```
then
```text
Operation: CloseContainer
Message: Cannot close container #67 while in UNHEALTHY state.
Result: CONTAINER_UNHEALTHY
```
By tracing the code, we can confirm why the chunk file was missing, because `FilePerBlockStrategy` intentionally skips empty chunk writes (this log was taken from a local reproduction run since this DEBUG log isn't showing up in the CI run log)
```text
2026-07-22 01:08:45,148 [e5111eb4-3873-4f6a-868f-2f14e06a9e36-ChunkReader-6] DEBUG impl.FilePerBlockStrategy (FilePerBlockStrategy.java:writeChunk(134)) - Skip writing empty chunk ChunkInfo{chunkName='116960482106540033.data.0, offset=0, len=0} in stage COMBINED
```
Therefore, when the test randomly selected zero, `WriteChunk` created no file, `PutBlock` failed to find it, and the datanode marked the container unhealthy. The final `CloseContainer` exception was a downstream symptom.
#### Proposed fix
The test previously generated a length in the range `[0, 1024)`, just make it non-empty `[1, 1024)`, would work.
As the test is intended to verify container state-machine idempotency, not empty-chunk behavior, so excluding zero keeps its randomized input within the valid scenario being tested.
## What is the link to the Apache JIRA
[Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull
request which starts with the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
If you do not have an ASF Jira account yet, please follow the first-time contributor
instructions in the [Jira guideline](../CONTRIBUTING.md#jira-guideline).
(Please replace this section with the link to the Apache JIRA)](https://issues.apache.org/jira/browse/HDDS-15930)
## How was this patch tested?
TBD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15930. Avoid zero-length chunk in TestContainerOperations [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr merged PR #10833:
URL: https://github.com/apache/ozone/pull/10833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15930. Avoid zero-length chunk in TestContainerOperations [ozone]
Posted by "henrybear327 (via GitHub)" <gi...@apache.org>.
henrybear327 commented on PR #10833:
URL: https://github.com/apache/ozone/pull/10833#issuecomment-5043950022
@chungen0126 I have attempted to enable the flaky-test-check workflow https://github.com/henrybear327/ozone/actions/runs/29906444081. Not sure if this is what you would like to get run :)
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15930. Avoid zero-length chunk in TestContainerOperations [ozone]
Posted by "henrybear327 (via GitHub)" <gi...@apache.org>.
henrybear327 commented on PR #10833:
URL: https://github.com/apache/ozone/pull/10833#issuecomment-5079227327
Hi @sarvekshayr ,
Attempted again
- https://github.com/henrybear327/ozone/actions/runs/30163847882
- https://github.com/henrybear327/ozone/actions/runs/30163852805
Hopefully this time it's done rightly :)
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15930. Avoid zero-length chunk in TestContainerOperations [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #10833:
URL: https://github.com/apache/ozone/pull/10833#issuecomment-5087639670
Thanks @henrybear327 for the patch, @anuragp010 and @chungen0126 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org