You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/08/21 14:03:42 UTC
[PR] CASSANDRA-21217: Stop decommission failing on a deleted hints file [cassandra]
1fanwang opened a new pull request, #5043:
URL: https://github.com/apache/cassandra/pull/5043
Decommission fails when hint transfer and expired-hints cleanup pick up the same file. The transfer opens a hints file cleanup has just unlinked, the failure propagates out of the decommission, and the node never leaves the ring. The reporter hit this three times and had to disable `transfer_hints_on_decommission`. After this patch the decommission completes with its hints transferred.
```
java.nio.file.NoSuchFileException: .../hints/5da9d583-...-1744041563101-2.hints
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:290)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
```
`HintsStore.poll()` is how a dispatch claims a descriptor. The removal is atomic, so one thread owns the file it points at, and the bulk delete path relies on that. `deleteExpiredHints` did not: it deleted each expired file in place and removed the descriptors afterwards, so between the expiry check and the unlink a dispatch could claim a descriptor whose file was already gone. The decommission transfer is the exposed caller, since it runs its dispatch tasks inline and never registers them for interruption.
The fix removes the descriptor from the queues before touching the file. A loser also drops the expiration it may have just cached, but leaves the dispatch offset alone since the new owner may be using it.
Warning and continuing on a missing file, as the ticket suggests, would hide data loss: a dispatch cannot tell a deleted expired file from a genuinely lost one.
## Testing
`HintsStoreCleanupRaceTest` uses Byteman to fix the interleaving rather than sleeps.
```
$ git checkout f8e3018 -- src/java/org/apache/cassandra/hints/HintsStore.java
$ ant test -Dtest.name=HintsStoreCleanupRaceTest -Dtest.timeout=480000
expiredHintsCleanupMustNotUnlinkAClaimedFile FAILED
Expired-hints cleanup unlinked cff39475-...-4.hints while a concurrent dispatch
had already claimed it.
tests="2" errors="0" failures="1"
$ git checkout HEAD -- src/java/org/apache/cassandra/hints/HintsStore.java
$ ant test -Dtest.name=HintsStoreCleanupRaceTest -Dtest.timeout=480000
tests="2" errors="0" failures="0"
```
The second test goes red only when the expiration cleanup is dropped, guarding a regression this patch would otherwise introduce.
The red run died on an unrelated initializer error in the unit JVM rather than the production `NoSuchFileException`, so the test asserts the ownership violation itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21217: Stop decommission failing on a deleted hints file [cassandra]
Posted by "1fanwang (via GitHub) via pr" <pr...@cassandra.apache.org>.
1fanwang commented on PR #5043:
URL: https://github.com/apache/cassandra/pull/5043#issuecomment-6806735575
@smiklosovic could you review when you have time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org