You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siddhant Sangwan (Jira)" <ji...@apache.org> on 2025/03/25 12:47:00 UTC

[jira] [Created] (HDDS-12691) Calculation of committed space in Datanode seems incorrect

Siddhant Sangwan created HDDS-12691:
---------------------------------------

             Summary: Calculation of committed space in Datanode seems incorrect
                 Key: HDDS-12691
                 URL: https://issues.apache.org/jira/browse/HDDS-12691
             Project: Apache Ozone
          Issue Type: Sub-task
          Components: Ozone Datanode
            Reporter: Siddhant Sangwan


The following method is called on every writeChunk request, after writing bytes to the disk to update stats:
{code}
  public void incrWriteBytes(long bytes) {
    long unused = getMaxSize() - getBytesUsed();

    this.writeBytes.addAndGet(bytes);
    /*
       Increase the cached Used Space in VolumeInfo as it
       maybe not updated, DU or DedicatedDiskSpaceUsage runs
       periodically to update the Used Space in VolumeInfo.
     */
    this.getVolume().incrementUsedSpace(bytes);
    // only if container size < max size
    if (committedSpace && unused > 0) {
      //with this write, container size might breach max size
      long decrement = Math.min(bytes, unused);
      this.getVolume().incCommittedBytes(0 - decrement);
    }
  }
{code}

For a container, 5GB (default) is committed when the container is created. Then every writeChunk is supposed to increment usedSpace and decrement committedSpace. Here, the logic of long decrement = Math.min(bytes, unused); seems incorrect to me.

Consider that the latest writeChunk wrote 256MB into a close-to-full container. Assume that now, unused = getMaxSize() - getBytesUsed() = 100MB. The current logic would decrement the committedSpace by 100MB, whereas it should have been decremented by 256MB.

Moreover, what about the case where the because of the latest chunk, the used size of the container exceeds the max size (used > max, for example I've seen 13GB containers in actual clusters since the max is just a soft limit), shouldn't we still decrement the committed space by getMaxSize() - (getBytesUsed() - bytes)? That's the remaining amount of committed space that this container was taking up in the volume before the "bytes" sized chunk was written to it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org