[jira] [Created] (HDDS-15711) Streaming Write Pipeline increment container blockCount twice

["Ivan Andika (Jira)" <ji...@apache.org>]

Ivan Andika created HDDS-15711:
----------------------------------

             Summary: Streaming Write Pipeline increment container blockCount twice
                 Key: HDDS-15711
                 URL: https://issues.apache.org/jira/browse/HDDS-15711
             Project: Apache Ozone
          Issue Type: Sub-task
            Reporter: Ivan Andika
            Assignee: Ivan Andika


Streaming close sends two EOF {{{}PutBlock{}}}s for the same block, while normal async write sends one.

Normal async path:
 * BlockOutputStream.executePutBlock only calls {{{}putBlockAsync(...){}}}.
 * Close with buffered data sends one EOF {{{}PutBlock{}}}.
 * Close after an earlier non-EOF putBlock sends one forced EOF {{{}PutBlock{}}}; the earlier non-EOF putBlock has already created the block / pending cache state, so EOF does not increment blockCount again.

Streaming path:
 * BlockDataStreamOutput.executePutBlock (line 425)] first sends an EOF putBlock through {{{}executePutBlockClose(...){}}}.
 * Then the same method also calls {{putBlockAsync(xceiverClient, blockData, close, token)}}
 * On the server, the data-stream close request is dispatched via {{{}STREAM_LINK{}}}, and the async request is dispatched as a normal {{{}PutBlock{}}}; both reach, then {{{}persistPutBlock{}}}.

So for a small streaming write that does not hit the intermediate data-stream flush boundary, close can race two EOF full-block commits for the same local ID. Both can see “block does not exist” before either batch commits, and both increment source {{{}blockCount{}}}.

I agree the broad write lock in {{persistPutBlock}} is too expensive for this. Better fix direction: remove that lock and make streaming close not submit two successful EOF {{{}PutBlock{}}}s for the same block. The cleanest target is probably {{{}BlockDataStreamOutput.executePutBlock{}}}: when {{{}close == true{}}}, use the data-stream close putBlock as the authoritative EOF commit and avoid the extra {{{}putBlockAsync{}}}, except for the existing fallback path when {{executePutBlockClose}} fails.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org