You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siddhant Sangwan (Jira)" <ji...@apache.org> on 2025/03/25 12:47:00 UTC
[jira] [Created] (HDDS-12691) Calculation of committed space in Datanode seems incorrect
Siddhant Sangwan created HDDS-12691:
---------------------------------------
Summary: Calculation of committed space in Datanode seems incorrect
Key: HDDS-12691
URL: https://issues.apache.org/jira/browse/HDDS-12691
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode
Reporter: Siddhant Sangwan
The following method is called on every writeChunk request, after writing bytes to the disk to update stats:
{code}
public void incrWriteBytes(long bytes) {
long unused = getMaxSize() - getBytesUsed();
this.writeBytes.addAndGet(bytes);
/*
Increase the cached Used Space in VolumeInfo as it
maybe not updated, DU or DedicatedDiskSpaceUsage runs
periodically to update the Used Space in VolumeInfo.
*/
this.getVolume().incrementUsedSpace(bytes);
// only if container size < max size
if (committedSpace && unused > 0) {
//with this write, container size might breach max size
long decrement = Math.min(bytes, unused);
this.getVolume().incCommittedBytes(0 - decrement);
}
}
{code}
For a container, 5GB (default) is committed when the container is created. Then every writeChunk is supposed to increment usedSpace and decrement committedSpace. Here, the logic of long decrement = Math.min(bytes, unused); seems incorrect to me.
Consider that the latest writeChunk wrote 256MB into a close-to-full container. Assume that now, unused = getMaxSize() - getBytesUsed() = 100MB. The current logic would decrement the committedSpace by 100MB, whereas it should have been decremented by 256MB.
Moreover, what about the case where the because of the latest chunk, the used size of the container exceeds the max size (used > max, for example I've seen 13GB containers in actual clusters since the max is just a soft limit), shouldn't we still decrement the committed space by getMaxSize() - (getBytesUsed() - bytes)? That's the remaining amount of committed space that this container was taking up in the volume before the "bytes" sized chunk was written to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org