You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Panagiotis Melidis via user <us...@cassandra.apache.org> on 2026/07/15 07:47:17 UTC
Recovery options for data written with far-future timestamps due to NTP clock jump
Hi all,
I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data.
If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys.
The only viable recovery approach at scale I've been able to identify is truncating the affected tables.
1. Are there any other known viable recovery procedures that don't require a full truncate?
2. Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios?
Any pointers to prior discussions, tooling, or patches would be greatly appreciated.
Thanks.
RE: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Panagiotis Melidis via user <us...@cassandra.apache.org>.
Hi all,
thank you for your replies.
@Yakir ntpd service is a prerequisite, but sometimes even NTP servers malfunction or misbehave if not properly configured with multiple redundant NTP upstream sources.
@Abe this is a fresh approach that I haven't thought about! However, I perceive the table migration as even more complex than manipulating the SStables offline. Depending on the cluster size and the complexity of the application logic, such migration can become impractical from both an operational and cost perspective (coordinating double writes/reads, verifying completeness of the backfill, determining the migration completion, managing the performance impact throughout, etc.).
@Jeff thank you for your replies.
CASSANDRA-18352 as a prevention measure looks promising, I am having a look.
CASSANDRA-12106 is very interesting, but also difficult to use with thousands or millions of affected partition keys. I'm primarily looking for a permanent recovery procedure.
Your proposal of implementing a custom mechanism for SStable scrub sounds like a viable alternative. I assume the same logic could be applied with offline SStable manipulation followed by sstableloader loading. Are there practical benefits of one approach over the other?
My main concern with either procedure is avoiding resurrection of the problematic cells. If we scrub nodes one at a time, we would need to disable auto-repairs and read-repair to prevent un-scrubbed replicas from re-propagating the future timestamps. Would we also need to discard pending hints generated before the scrub? Are there any other precautionary measures needed to ensure the scrubbed data doesn't get overwritten by stale replicas?
Regards,
Panagiotis
-----Original Message-----
From: Jeff Jirsa <jj...@gmail.com>
Sent: Wednesday, July 15, 2026 9:28 PM
To: cassandra <us...@cassandra.apache.org>
Subject: Re: Recovery options for data written with far-future timestamps due to NTP clock jump
That’s a good point. CASSANDRA-12106 would let you blacklist those partitions if you know they’re invalid. That’s probably strictly easier than truncating the table.
> On Jul 15, 2026, at 9:01 AM, Abe Ratnofsky <ar...@fastmail.com> wrote:
>
> Read repair will propagate the future-writes, so if you scrub them from one replica and have concurrent reads, they can re-appear.
>
> Depending on your requirements you might also be able to post-filter the future-writes at query time, by including SELECT WRITETIME(col) in your read queries and skipping them.
>
> On Wed, Jul 15, 2026, at 8:39 AM, Jeff Jirsa wrote:
>> Double writing / double reading gets messy because it requires an
>> awful lot of work to ensure it’s correct as you move data around.
>>
>> Just fixing it during compaction or scrub doesnt require app changes,
>> just disable repair, start rewriting the sstables, enable repair
>> again when you’re done.
>>
>> Especially if 18352 gives you a cassandra-internal “is this a legal
>> timestamp” predicate, that seems way easier than changing all the app
>> logic.
>>
>>
>>> On Jul 15, 2026, at 8:21 AM, Abe Ratnofsky <ab...@apache.org> wrote:
>>>
>>> For recovery, I’d recommend approaching this as a regular table migration, where you start writing to a new table and backfill using cassandra-analytics, filtering out all cells with WRITETIME in the distant future.
>>>
>>> Freeze your backups so they don’t get purged. You might be able to recover some overwritten values from backup SSTables if they weren’t compacted yet.
>>>
>>> On Wed, Jul 15, 2026, at 8:16 AM, Jeff Jirsa wrote:
>>>> It’s not that it’s not running, it’s that ntpd in startup phases
>>>> can have big steps in some configs. Also alert on large clock
>>>> drift. If you run cassandra, you need alerts on ntpd
>>>>
>>>>> On Jul 15, 2026, at 8:13 AM, Yakir Gibraltar <ya...@gmail.com> wrote:
>>>>>
>>>>> Configure Cassandra to start after ntpd service , easy on systemd.
>>>>>
>>>>> Thank you, Yakir Gibraltar
>>>>>
>>>>> On Wed, Jul 15, 2026, 10:47 Panagiotis Melidis via user <us...@cassandra.apache.org> wrote:
>>>>>> Hi all,____
>>>>>> __ __
>>>>>> I'm looking for guidance on a scenario involving NTP clock skew
>>>>>> and its impact on Cassandra data.____ __ __ If an NTP malfunction
>>>>>> causes the system clock on multiple Cassandra nodes to jump far
>>>>>> into the future (e.g., 100 years), any writes processed during
>>>>>> that window will carry far-future timestamps. Once the clock is
>>>>>> corrected, these entries effectively become permanent and shadow
>>>>>> any new write/delete in the affected partition keys.____ __ __
>>>>>> The only viable recovery approach at scale I've been able to
>>>>>> identify is truncating the affected tables.____ 1. Are there any
>>>>>> other known viable recovery procedures that don't require a full
>>>>>> truncate?____ 2. Are there any known existing or planned
>>>>>> preventive measures in Cassandra to guard against such
>>>>>> scenarios?____ __ __ Any pointers to prior discussions, tooling,
>>>>>> or patches would be greatly appreciated.____ __ __ Thanks.____
Re: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Jeff Jirsa <jj...@gmail.com>.
That’s a good point. CASSANDRA-12106 would let you blacklist those partitions if you know they’re invalid. That’s probably strictly easier than truncating the table.
> On Jul 15, 2026, at 9:01 AM, Abe Ratnofsky <ar...@fastmail.com> wrote:
>
> Read repair will propagate the future-writes, so if you scrub them from one replica and have concurrent reads, they can re-appear.
>
> Depending on your requirements you might also be able to post-filter the future-writes at query time, by including SELECT WRITETIME(col) in your read queries and skipping them.
>
> On Wed, Jul 15, 2026, at 8:39 AM, Jeff Jirsa wrote:
>> Double writing / double reading gets messy because it requires an awful
>> lot of work to ensure it’s correct as you move data around.
>>
>> Just fixing it during compaction or scrub doesnt require app changes,
>> just disable repair, start rewriting the sstables, enable repair again
>> when you’re done.
>>
>> Especially if 18352 gives you a cassandra-internal “is this a legal
>> timestamp” predicate, that seems way easier than changing all the app
>> logic.
>>
>>
>>> On Jul 15, 2026, at 8:21 AM, Abe Ratnofsky <ab...@apache.org> wrote:
>>>
>>> For recovery, I’d recommend approaching this as a regular table migration, where you start writing to a new table and backfill using cassandra-analytics, filtering out all cells with WRITETIME in the distant future.
>>>
>>> Freeze your backups so they don’t get purged. You might be able to recover some overwritten values from backup SSTables if they weren’t compacted yet.
>>>
>>> On Wed, Jul 15, 2026, at 8:16 AM, Jeff Jirsa wrote:
>>>> It’s not that it’s not running, it’s that ntpd in startup phases can
>>>> have big steps in some configs. Also alert on large clock drift. If you
>>>> run cassandra, you need alerts on ntpd
>>>>
>>>>> On Jul 15, 2026, at 8:13 AM, Yakir Gibraltar <ya...@gmail.com> wrote:
>>>>>
>>>>> Configure Cassandra to start after ntpd service , easy on systemd.
>>>>>
>>>>> Thank you, Yakir Gibraltar
>>>>>
>>>>> On Wed, Jul 15, 2026, 10:47 Panagiotis Melidis via user <us...@cassandra.apache.org> wrote:
>>>>>> Hi all,____
>>>>>> __ __
>>>>>> I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data.____
>>>>>> __ __
>>>>>> If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys.____
>>>>>> __ __
>>>>>> The only viable recovery approach at scale I've been able to identify is truncating the affected tables.____
>>>>>> 1. Are there any other known viable recovery procedures that don't require a full truncate?____
>>>>>> 2. Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios?____
>>>>>> __ __
>>>>>> Any pointers to prior discussions, tooling, or patches would be greatly appreciated.____
>>>>>> __ __
>>>>>> Thanks.____
Re: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Abe Ratnofsky <ar...@fastmail.com>.
Read repair will propagate the future-writes, so if you scrub them from one replica and have concurrent reads, they can re-appear.
Depending on your requirements you might also be able to post-filter the future-writes at query time, by including SELECT WRITETIME(col) in your read queries and skipping them.
On Wed, Jul 15, 2026, at 8:39 AM, Jeff Jirsa wrote:
> Double writing / double reading gets messy because it requires an awful
> lot of work to ensure it’s correct as you move data around.
>
> Just fixing it during compaction or scrub doesnt require app changes,
> just disable repair, start rewriting the sstables, enable repair again
> when you’re done.
>
> Especially if 18352 gives you a cassandra-internal “is this a legal
> timestamp” predicate, that seems way easier than changing all the app
> logic.
>
>
>> On Jul 15, 2026, at 8:21 AM, Abe Ratnofsky <ab...@apache.org> wrote:
>>
>> For recovery, I’d recommend approaching this as a regular table migration, where you start writing to a new table and backfill using cassandra-analytics, filtering out all cells with WRITETIME in the distant future.
>>
>> Freeze your backups so they don’t get purged. You might be able to recover some overwritten values from backup SSTables if they weren’t compacted yet.
>>
>> On Wed, Jul 15, 2026, at 8:16 AM, Jeff Jirsa wrote:
>>> It’s not that it’s not running, it’s that ntpd in startup phases can
>>> have big steps in some configs. Also alert on large clock drift. If you
>>> run cassandra, you need alerts on ntpd
>>>
>>>> On Jul 15, 2026, at 8:13 AM, Yakir Gibraltar <ya...@gmail.com> wrote:
>>>>
>>>> Configure Cassandra to start after ntpd service , easy on systemd.
>>>>
>>>> Thank you, Yakir Gibraltar
>>>>
>>>> On Wed, Jul 15, 2026, 10:47 Panagiotis Melidis via user <us...@cassandra.apache.org> wrote:
>>>>> Hi all,____
>>>>> __ __
>>>>> I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data.____
>>>>> __ __
>>>>> If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys.____
>>>>> __ __
>>>>> The only viable recovery approach at scale I've been able to identify is truncating the affected tables.____
>>>>> 1. Are there any other known viable recovery procedures that don't require a full truncate?____
>>>>> 2. Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios?____
>>>>> __ __
>>>>> Any pointers to prior discussions, tooling, or patches would be greatly appreciated.____
>>>>> __ __
>>>>> Thanks.____
Re: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Jeff Jirsa <jj...@gmail.com>.
Double writing / double reading gets messy because it requires an awful lot of work to ensure it’s correct as you move data around.
Just fixing it during compaction or scrub doesnt require app changes, just disable repair, start rewriting the sstables, enable repair again when you’re done.
Especially if 18352 gives you a cassandra-internal “is this a legal timestamp” predicate, that seems way easier than changing all the app logic.
> On Jul 15, 2026, at 8:21 AM, Abe Ratnofsky <ab...@apache.org> wrote:
>
> For recovery, I’d recommend approaching this as a regular table migration, where you start writing to a new table and backfill using cassandra-analytics, filtering out all cells with WRITETIME in the distant future.
>
> Freeze your backups so they don’t get purged. You might be able to recover some overwritten values from backup SSTables if they weren’t compacted yet.
>
> On Wed, Jul 15, 2026, at 8:16 AM, Jeff Jirsa wrote:
>> It’s not that it’s not running, it’s that ntpd in startup phases can
>> have big steps in some configs. Also alert on large clock drift. If you
>> run cassandra, you need alerts on ntpd
>>
>>> On Jul 15, 2026, at 8:13 AM, Yakir Gibraltar <ya...@gmail.com> wrote:
>>>
>>> Configure Cassandra to start after ntpd service , easy on systemd.
>>>
>>> Thank you, Yakir Gibraltar
>>>
>>> On Wed, Jul 15, 2026, 10:47 Panagiotis Melidis via user <us...@cassandra.apache.org> wrote:
>>>> Hi all,____
>>>> __ __
>>>> I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data.____
>>>> __ __
>>>> If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys.____
>>>> __ __
>>>> The only viable recovery approach at scale I've been able to identify is truncating the affected tables.____
>>>> 1. Are there any other known viable recovery procedures that don't require a full truncate?____
>>>> 2. Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios?____
>>>> __ __
>>>> Any pointers to prior discussions, tooling, or patches would be greatly appreciated.____
>>>> __ __
>>>> Thanks.____
Re: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Abe Ratnofsky <ab...@apache.org>.
For recovery, I’d recommend approaching this as a regular table migration, where you start writing to a new table and backfill using cassandra-analytics, filtering out all cells with WRITETIME in the distant future.
Freeze your backups so they don’t get purged. You might be able to recover some overwritten values from backup SSTables if they weren’t compacted yet.
On Wed, Jul 15, 2026, at 8:16 AM, Jeff Jirsa wrote:
> It’s not that it’s not running, it’s that ntpd in startup phases can
> have big steps in some configs. Also alert on large clock drift. If you
> run cassandra, you need alerts on ntpd
>
>> On Jul 15, 2026, at 8:13 AM, Yakir Gibraltar <ya...@gmail.com> wrote:
>>
>> Configure Cassandra to start after ntpd service , easy on systemd.
>>
>> Thank you, Yakir Gibraltar
>>
>> On Wed, Jul 15, 2026, 10:47 Panagiotis Melidis via user <us...@cassandra.apache.org> wrote:
>>> Hi all,____
>>> __ __
>>> I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data.____
>>> __ __
>>> If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys.____
>>> __ __
>>> The only viable recovery approach at scale I've been able to identify is truncating the affected tables.____
>>> 1. Are there any other known viable recovery procedures that don't require a full truncate?____
>>> 2. Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios?____
>>> __ __
>>> Any pointers to prior discussions, tooling, or patches would be greatly appreciated.____
>>> __ __
>>> Thanks.____
Re: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Jeff Jirsa <jj...@gmail.com>.
Also note this guardrail from CASSANDRA-18352 <https://issues.apache.org/jira/browse/CASSANDRA-18352>
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/guardrails/Guardrails.java#L632
cassandra/src/java/org/apache/cassandra/db/guardrails/Guardrails.java at trunk · apache/cassandra
github.com
> On Jul 15, 2026, at 8:16 AM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> It’s not that it’s not running, it’s that ntpd in startup phases can have big steps in some configs. Also alert on large clock drift. If you run cassandra, you need alerts on ntpd
>
>> On Jul 15, 2026, at 8:13 AM, Yakir Gibraltar <ya...@gmail.com> wrote:
>>
>> Configure Cassandra to start after ntpd service , easy on systemd.
>>
>> Thank you, Yakir Gibraltar
>>
>> On Wed, Jul 15, 2026, 10:47 Panagiotis Melidis via user <user@cassandra.apache.org <ma...@cassandra.apache.org>> wrote:
>>> Hi all,
>>>
>>>
>>>
>>> I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data.
>>>
>>>
>>>
>>> If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys.
>>>
>>>
>>>
>>> The only viable recovery approach at scale I've been able to identify is truncating the affected tables.
>>>
>>> Are there any other known viable recovery procedures that don't require a full truncate?
>>> Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios?
>>>
>>>
>>> Any pointers to prior discussions, tooling, or patches would be greatly appreciated.
>>>
>>>
>>>
>>> Thanks.
>>>
>
Re: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Jeff Jirsa <jj...@gmail.com>.
It’s not that it’s not running, it’s that ntpd in startup phases can have big steps in some configs. Also alert on large clock drift. If you run cassandra, you need alerts on ntpd
> On Jul 15, 2026, at 8:13 AM, Yakir Gibraltar <ya...@gmail.com> wrote:
>
> Configure Cassandra to start after ntpd service , easy on systemd.
>
> Thank you, Yakir Gibraltar
>
> On Wed, Jul 15, 2026, 10:47 Panagiotis Melidis via user <user@cassandra.apache.org <ma...@cassandra.apache.org>> wrote:
>> Hi all,
>>
>>
>>
>> I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data.
>>
>>
>>
>> If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys.
>>
>>
>>
>> The only viable recovery approach at scale I've been able to identify is truncating the affected tables.
>>
>> Are there any other known viable recovery procedures that don't require a full truncate?
>> Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios?
>>
>>
>> Any pointers to prior discussions, tooling, or patches would be greatly appreciated.
>>
>>
>>
>> Thanks.
>>
Re: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Yakir Gibraltar <ya...@gmail.com>.
Configure Cassandra to start after ntpd service , easy on systemd.
Thank you, Yakir Gibraltar
On Wed, Jul 15, 2026, 10:47 Panagiotis Melidis via user <
user@cassandra.apache.org> wrote:
> Hi all,
>
>
>
> I'm looking for guidance on a scenario involving NTP clock skew and its
> impact on Cassandra data.
>
>
>
> If an NTP malfunction causes the system clock on multiple Cassandra nodes
> to jump far into the future (e.g., 100 years), any writes processed during
> that window will carry far-future timestamps. Once the clock is corrected,
> these entries effectively become permanent and shadow any new write/delete
> in the affected partition keys.
>
>
>
> The only viable recovery approach at scale I've been able to identify is
> truncating the affected tables.
>
> 1. Are there any other known viable recovery procedures that don't
> require a full truncate?
> 2. Are there any known existing or planned preventive measures in
> Cassandra to guard against such scenarios?
>
>
>
> Any pointers to prior discussions, tooling, or patches would be greatly
> appreciated.
>
>
>
> Thanks.
>
RE: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Panagiotis Melidis via user <us...@cassandra.apache.org>.
Hi Henry,
thank you for the detailed response.
I agree that full truncation is currently the only operationally safe recovery at scale with SSTable surgery as a distant second given its complexity and error-prone nature.
On the second recovery option (dropping and recreating the affected partitions), it should be possible to “ignore” logically problematic partitions keys on the application side, but as the problematic partition keys increase, so does the complexity of implementing this. Rewriting fresh data over the problematic partition keys is not straightforward, because the client would need to start writing with a timestamp that is even further into the future, which compounds the original problem.
On the prevention measures, I agree that the first line of defense should be OS-level protections and then Cassandra-level protections. In my understanding, client-side timestamps cannot prevent such issues, since tombstones use two fields: marked_deleted and local_delete_time. The first can be set if the client sets a writetime (using timestamp) and is used for conflict resolution between rows based on Last-Write-Wins. The second value is set by the Cassandra node based on the local clock and is used to determine when a row can be removed via compaction. The client has no control over local_delete_time, so it is not possible to enforce sanity checks on the client-side or cleanup the problematic rows.
Regards,
Panagiotis
From: Henry Pan (HP) <he...@gmail.com>
Sent: Wednesday, July 15, 2026 6:02 PM
To: user@cassandra.apache.org
Cc: Panagiotis Melidis <pa...@ericsson.com>
Subject: Re: Recovery options for data written with far-future timestamps due to NTP clock jump
You don't often get email from henrypan1@gmail.com<ma...@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Here’s a clean, technically solid reply you can send back to Panagiotis — one that reflects how senior Cassandra engineers typically answer this class of question, and gives you credibility without over‑committing.
Reply Draft
Hi Panagiotis,
Clock jumps of that magnitude create exactly the failure mode you described — “immortal” cells whose timestamps sit far in the future and permanently shadow any subsequent writes or deletes. Once those timestamps are written to SSTables, Cassandra’s MVCC model treats them as authoritative, and normal repair/compaction cannot unwind them.
Below is the current state of what’s realistically possible.
1. Recovery options (beyond full truncate)
Unfortunately, there is no general-purpose, cluster‑safe recovery mechanism that can selectively unwind far‑future timestamps. The viable options depend on how localized the damage is:
* Targeted SSTable surgery (offline) Possible only if the affected partitions are known and limited. You can use sstablemetadata / sstabledump to identify SSTables containing future timestamps, then:
* Extract the partitions
* Rewrite timestamps to sane values
* Rebuild SSTables with sstableloader This is extremely manual, error‑prone, and not recommended at scale.
* Drop + recreate affected partitions (application-level) If the impacted keys are identifiable, you can delete them by:
* Dropping the partition key logically in the application
* Rewriting fresh data This works only if the application can tolerate losing historical data for those keys.
* Full table truncate This is the only cluster‑safe and operationally reliable remediation when the corruption is widespread. It guarantees removal of all future‑timestamp cells and avoids partial resurrection of bad data.
There is no built‑in Cassandra mechanism to “roll back” timestamps or override MVCC ordering once bad timestamps exist on disk.
2. Preventive measures (existing and planned)
Cassandra historically assumes monotonic clocks and does not defend against catastrophic NTP failures. However, there are several layers of mitigation you can implement:
* JVM‑level time fencing Use -Dcassandra.max_hint_window and -Dcassandra.write_request_timeout_in_ms defensively, but these do not prevent future timestamps.
* OS‑level protections
* Disable step corrections in NTP/chrony
* Require slew‑only adjustments
* Enforce max‑offset thresholds These are the most effective real‑world protections.
* Cassandra‑side proposals There have been discussions in the dev list about:
* Rejecting writes with timestamps beyond a configurable horizon
* Emitting warnings or failing requests when clock skew exceeds a threshold
* Adding a “timestamp quarantine” mode None of these have been merged into trunk yet.
* Client‑side timestamps If your workload uses client‑supplied timestamps, you can enforce sanity checks before sending writes. This does not help when server‑side timestamps are used.
3. Prior discussions / references
This failure mode has come up in several JIRA threads and mailing list discussions over the years, typically under “future timestamps”, “clock skew”, or “NTP step jumps”. The consensus has always been:
* Once future timestamps are written, Cassandra will honor them forever
* There is no safe automated rollback
* Prevention is the only reliable strategy
Summary
Your assessment is correct: if multiple nodes wrote far‑future timestamps, a full truncate is the only safe cluster‑wide recovery. Partial repair is possible only when the affected partitions are few and manually identifiable. Preventive measures must be implemented at the OS/NTP layer, as Cassandra currently does not enforce timestamp sanity.
Happy to dig deeper into any of the above if useful.
If you want, I can also prepare a shorter version tailored for the Cassandra user mailing list tone, or a more formal one suitable for a JIRA ticket.
Thanks & Best Regards
Henry PAN
Sr. Lead Cloud Architect
(425) 802--3975
https://www.linkedin.com/in/henrypan1
On Wed, Jul 15, 2026 at 7:35 AM Jeff Jirsa <jj...@gmail.com>> wrote:
This is actually both relatively hard and also potentially straightforward to resolve
It’s hard because you’re going to have to write code because Cassandra can’t do this without customization.
It’s straightforward though
You have to add a mechanism to sstable scrub that looks for known unsafe timestamps (in the future more than X years) and scrubs them to “now”
You’ll want to be safe against the micros/millis difference to pay attention to that.
You’ll obviously want to test this extensively before running it in real life
On Jul 15, 2026, at 12:47 AM, Panagiotis Melidis via user <us...@cassandra.apache.org>> wrote:
Hi all,
I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data.
If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys.
The only viable recovery approach at scale I've been able to identify is truncating the affected tables.
1. Are there any other known viable recovery procedures that don't require a full truncate?
2. Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios?
Any pointers to prior discussions, tooling, or patches would be greatly appreciated.
Thanks.
Re: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by "Henry Pan (HP)" <he...@gmail.com>.
Here’s a clean, technically solid reply you can send back to Panagiotis —
one that reflects how senior Cassandra engineers typically answer this
class of question, and gives you credibility without over‑committing.
*Reply Draft*
Hi Panagiotis,
Clock jumps of that magnitude create exactly the failure mode you described
— “immortal” cells whose timestamps sit far in the future and permanently
shadow any subsequent writes or deletes. Once those timestamps are written
to SSTables, Cassandra’s MVCC model treats them as authoritative, and
normal repair/compaction cannot unwind them.
Below is the current state of what’s realistically possible.
*1. Recovery options (beyond full truncate)*
Unfortunately, there is no general-purpose, cluster‑safe recovery mechanism
that can selectively unwind far‑future timestamps. The viable options
depend on how localized the damage is:
-
*Targeted SSTable surgery (offline)* Possible only if the affected
partitions are known and limited. You can use sstablemetadata /
sstabledump to identify SSTables containing future timestamps, then:
-
Extract the partitions
-
Rewrite timestamps to sane values
-
Rebuild SSTables with sstableloader This is extremely manual,
error‑prone, and not recommended at scale.
-
*Drop + recreate affected partitions (application-level)* If the
impacted keys are identifiable, you can delete them by:
-
Dropping the partition key logically in the application
-
Rewriting fresh data This works only if the application can tolerate
losing historical data for those keys.
-
*Full table truncate* This is the only *cluster‑safe* and *operationally
reliable* remediation when the corruption is widespread. It guarantees
removal of all future‑timestamp cells and avoids partial resurrection of
bad data.
There is no built‑in Cassandra mechanism to “roll back” timestamps or
override MVCC ordering once bad timestamps exist on disk.
*2. Preventive measures (existing and planned)*
Cassandra historically assumes monotonic clocks and does not defend against
catastrophic NTP failures. However, there are several layers of mitigation
you can implement:
-
*JVM‑level time fencing* Use -Dcassandra.max_hint_window and
-Dcassandra.write_request_timeout_in_ms defensively, but these do not
prevent future timestamps.
-
*OS‑level protections*
-
Disable step corrections in NTP/chrony
-
Require slew‑only adjustments
-
Enforce max‑offset thresholds These are the most effective real‑world
protections.
-
*Cassandra‑side proposals* There have been discussions in the dev list
about:
-
Rejecting writes with timestamps beyond a configurable horizon
-
Emitting warnings or failing requests when clock skew exceeds a
threshold
-
Adding a “timestamp quarantine” mode None of these have been merged
into trunk yet.
-
*Client‑side timestamps* If your workload uses client‑supplied
timestamps, you can enforce sanity checks before sending writes. This does
not help when server‑side timestamps are used.
*3. Prior discussions / references*
This failure mode has come up in several JIRA threads and mailing list
discussions over the years, typically under “future timestamps”, “clock
skew”, or “NTP step jumps”. The consensus has always been:
-
Once future timestamps are written, Cassandra will honor them forever
-
There is no safe automated rollback
-
Prevention is the only reliable strategy
*Summary*
Your assessment is correct: if multiple nodes wrote far‑future timestamps,
a full truncate is the only safe cluster‑wide recovery. Partial repair is
possible only when the affected partitions are few and manually
identifiable. Preventive measures must be implemented at the OS/NTP layer,
as Cassandra currently does not enforce timestamp sanity.
Happy to dig deeper into any of the above if useful.
If you want, I can also prepare a shorter version tailored for the
Cassandra user mailing list tone, or a more formal one suitable for a JIRA
ticket.
Thanks & Best Regards
Henry PAN
Sr. Lead Cloud Architect
(425) 802--3975
https://www.linkedin.com/in/henrypan1
On Wed, Jul 15, 2026 at 7:35 AM Jeff Jirsa <jj...@gmail.com> wrote:
> This is actually both relatively hard and also potentially straightforward
> to resolve
>
> It’s hard because you’re going to have to write code because Cassandra
> can’t do this without customization.
>
> It’s straightforward though
>
> You have to add a mechanism to sstable scrub that looks for known unsafe
> timestamps (in the future more than X years) and scrubs them to “now”
>
> You’ll want to be safe against the micros/millis difference to pay
> attention to that.
>
> You’ll obviously want to test this extensively before running it in real
> life
>
>
>
> On Jul 15, 2026, at 12:47 AM, Panagiotis Melidis via user <
> user@cassandra.apache.org> wrote:
>
>
>
> Hi all,
>
>
>
> I'm looking for guidance on a scenario involving NTP clock skew and its
> impact on Cassandra data.
>
>
>
> If an NTP malfunction causes the system clock on multiple Cassandra nodes
> to jump far into the future (e.g., 100 years), any writes processed during
> that window will carry far-future timestamps. Once the clock is corrected,
> these entries effectively become permanent and shadow any new write/delete
> in the affected partition keys.
>
>
>
> The only viable recovery approach at scale I've been able to identify is
> truncating the affected tables.
>
> 1. Are there any other known viable recovery procedures that don't
> require a full truncate?
> 2. Are there any known existing or planned preventive measures in
> Cassandra to guard against such scenarios?
>
>
>
> Any pointers to prior discussions, tooling, or patches would be greatly
> appreciated.
>
>
>
> Thanks.
>
>
Re: Recovery options for data written with far-future timestamps due to NTP clock jump
Posted by Jeff Jirsa <jj...@gmail.com>.
This is actually both relatively hard and also potentially straightforward to resolve
It’s hard because you’re going to have to write code because Cassandra can’t do this without customization.
It’s straightforward though
You have to add a mechanism to sstable scrub that looks for known unsafe timestamps (in the future more than X years) and scrubs them to “now”
You’ll want to be safe against the micros/millis difference to pay attention to that.
You’ll obviously want to test this extensively before running it in real life
> On Jul 15, 2026, at 12:47 AM, Panagiotis Melidis via user <us...@cassandra.apache.org> wrote:
>
>
> Hi all,
>
> I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data.
>
> If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys.
>
> The only viable recovery approach at scale I've been able to identify is truncating the affected tables.
> Are there any other known viable recovery procedures that don't require a full truncate?
> Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios?
>
> Any pointers to prior discussions, tooling, or patches would be greatly appreciated.
>
> Thanks.