You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sarvekshayr (via GitHub)" <gi...@apache.org> on 2026/01/21 08:28:52 UTC
[PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
sarvekshayr opened a new pull request, #9655:
URL: https://github.com/apache/ozone/pull/9655
## What changes were proposed in this pull request?
Saw this warning when datanode disk was nearly full:
```
2025-12-15 10:37:20,903 WARN [166c5ca8-343e-46ed-b619-84ac193e0069-ChunkReader-215]-org.apache.hadoop.hdds.fs.CachingSpaceUsageSource: Attempted to decrement available space to a negative value. Current: 0, Decrement: 1048576, Source: /ipdr_ozone31/hadoop-ozone/datanode/data
```
Prior to this message, there were many failed writes. Perhaps it needs to increment the value when the write fails.
The fix adds rollback logic in `KeyValueHandler.handleWriteChunk()` that tracks when a chunk write succeeds and increments the `usedSpace` counter. If any subsequent operation fails, the exception handler calls `volume.decrementUsedSpace()` to restore the counter.
## What is the link to the Apache JIRA
[HDDS-14183](https://issues.apache.org/jira/browse/HDDS-14183)
## How was this patch tested?
CI: https://github.com/sarvekshayr/ozone/actions/runs/21200210393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "sreejasahithi (via GitHub)" <gi...@apache.org>.
sreejasahithi commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2721024693
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
Review Comment:
Could you please add test coverage to verify the restore of both usedspace and committedBytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9655:
URL: https://github.com/apache/ozone/pull/9655#issuecomment-4114501634
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2721718934
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1118,31 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
}
+ /**
+ * Commit space reserved for write to usedSpace when write operation succeeds.
+ */
+ private void commitSpaceReservedForWrite(HddsVolume volume, boolean spaceReserved, long bytes) {
+ if (spaceReserved) {
+ volume.releaseReservedSpaceForWrite(bytes);
+ volume.incrementUsedSpace(bytes);
Review Comment:
since release and increment are not atomic, its better to incrementUsedSpace first and then release reserved space, if its done as it currently implemented if other thread check free space between these 2 calls it could try to use free space that already used by this write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "sreejasahithi (via GitHub)" <gi...@apache.org>.
sreejasahithi commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2715574243
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1109,19 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
}
+ /**
+ * Roll back usedSpace when write operation fails after writeChunk succeeded.
+ */
+ private void rollbackUsedSpaceOnWriteFailure(KeyValueContainer kvContainer, boolean writeChunkSucceeded,
Review Comment:
When write succeeds, `incrWriteBytes()` decrements committedBytes (in ContainerData.java), but rollback only restores usedSpace. Should committedBytes also be rolled back to maintain consistency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2734953959
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1118,31 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
}
+ /**
+ * Commit space reserved for write to usedSpace when write operation succeeds.
+ */
+ private void commitSpaceReservedForWrite(HddsVolume volume, boolean spaceReserved, long bytes) {
Review Comment:
Updated the logic to make the flow clearer by moving the `spaceReserved` check to the call site. This makes it explicit that the commit operation is invoked only when space has actually been reserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9655:
URL: https://github.com/apache/ozone/pull/9655#issuecomment-3923932051
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2712806865
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1109,19 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
}
+ /**
+ * Roll back usedSpace when write operation fails after writeChunk succeeded.
+ */
+ private void rollbackUsedSpaceOnWriteFailure(KeyValueContainer kvContainer, boolean writeChunkSucceeded,
+ long bytesWritten) {
+ if (writeChunkSucceeded && bytesWritten > 0) {
+ HddsVolume volume = kvContainer.getContainerData().getVolume();
+ if (volume != null) {
+ volume.decrementUsedSpace(bytesWritten);
Review Comment:
Can logic to be changed to update volume's used space when data is successfully written to the disk. if we want to track space that would be used by concurrent writes we can introduce the concept of space reserved for in-flight writes and update usedSpace only if write is successful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2721697483
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1118,31 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
Review Comment:
what is happening with reserved space when write is completed?
Typically when write successful reserved space is released and used space is increased by the write size. And while write is the flight free space is calculated as "total - used - reserved"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2993548647
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java:
##########
@@ -672,6 +674,14 @@ public synchronized void updateWrite(long length, boolean overwrite) {
writeBytes += length;
}
+ /**
+ * Increment blockBytes by the given delta.
+ * This is used for overwrite operations that extend the file.
+ */
+ public synchronized void incrementBlockBytes(long delta) {
+ blockBytes += delta;
Review Comment:
`writeCount` and `writeBytes` were being incremented twice when overwrite=true.
Once updateWrite() would increment and then again in updateWriteStats().
```
if (overwrite) {
long fileLengthAfterWrite = offset + chunkLength;
if (fileLengthAfterWrite > fileLengthBeforeWrite) {
containerData.getStatistics().updateWrite(fileLengthAfterWrite - fileLengthBeforeWrite, false);
}
}
containerData.updateWriteStats(chunkLength, overwrite);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2716353344
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1109,19 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
}
+ /**
+ * Roll back usedSpace when write operation fails after writeChunk succeeded.
+ */
+ private void rollbackUsedSpaceOnWriteFailure(KeyValueContainer kvContainer, boolean writeChunkSucceeded,
+ long bytesWritten) {
+ if (writeChunkSucceeded && bytesWritten > 0) {
+ HddsVolume volume = kvContainer.getContainerData().getVolume();
+ if (volume != null) {
+ volume.decrementUsedSpace(bytesWritten);
Review Comment:
I've introduced a variable to track the space reserved for in-flight writes.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1109,19 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
}
+ /**
+ * Roll back usedSpace when write operation fails after writeChunk succeeded.
+ */
+ private void rollbackUsedSpaceOnWriteFailure(KeyValueContainer kvContainer, boolean writeChunkSucceeded,
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2992305307
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java:
##########
@@ -672,6 +674,14 @@ public synchronized void updateWrite(long length, boolean overwrite) {
writeBytes += length;
}
+ /**
+ * Increment blockBytes by the given delta.
+ * This is used for overwrite operations that extend the file.
+ */
+ public synchronized void incrementBlockBytes(long delta) {
+ blockBytes += delta;
Review Comment:
what is the impact of not updating below as earlier before this method used to happen,
writeCount++;
writeBytes += length;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #9655:
URL: https://github.com/apache/ozone/pull/9655
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #9655:
URL: https://github.com/apache/ozone/pull/9655#issuecomment-4134205338
Thanks @sumitagrawl, @siddhantsangwan, @yandrey321 and @sreejasahithi for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2721697483
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1118,31 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
Review Comment:
what is happening with reserved space when write is completed?
Typically when write successful reserved space is released and used space is increased by the write size. And while write is the flight free space is calculated as "total - used - reserved"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2721706894
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1118,31 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
}
+ /**
+ * Commit space reserved for write to usedSpace when write operation succeeds.
+ */
+ private void commitSpaceReservedForWrite(HddsVolume volume, boolean spaceReserved, long bytes) {
Review Comment:
can commit happen when space is not reserved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2721718934
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1102,6 +1118,31 @@ ContainerCommandResponseProto handleWriteChunk(
return getWriteChunkResponseSuccess(request, blockDataProto);
}
+ /**
+ * Commit space reserved for write to usedSpace when write operation succeeds.
+ */
+ private void commitSpaceReservedForWrite(HddsVolume volume, boolean spaceReserved, long bytes) {
+ if (spaceReserved) {
+ volume.releaseReservedSpaceForWrite(bytes);
+ volume.incrementUsedSpace(bytes);
Review Comment:
since release and increment are not inside critical section, its better to incrementUsedSpace first and then release reserved space, if its done as it currently implemented if other thread check free space between these 2 calls it could try to use free space that already used by this write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14183. Attempted to decrement available space to a negative value [ozone]
Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #9655:
URL: https://github.com/apache/ozone/pull/9655#discussion_r2870616088
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/FilePerBlockStrategy.java:
##########
Review Comment:
Yes, but in the overwrite case it's getting called once at line 191 and again at line 195.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org