You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "pmcfadin (via GitHub)" <gi...@apache.org> on 2026/09/08 03:34:41 UTC
[PR] CASSANDRA-21664: Make schema changes cost O(1) in the number of tables and fail loudly at schema limits [cassandra]
pmcfadin opened a new pull request, #5128:
URL: https://github.com/apache/cassandra/pull/5128
Supersedes #5121, #5122, #5123 (their commits are the first three here, unchanged).
Branch: pmcfadin/schema-at-scale, 10 commits, rebased on cassandra-6.0.
## What this fixes
Creating thousands of tables is slow for two independent reasons and unsafe for a third:
1. Every CREATE TABLE synchronously flushes all system_schema tables on the log follower (92 of ~121 ms).
2. Several per-epoch operations walk or rebuild the whole schema (diff, keyspace map, table map, metrics array).
3. Past ~31k minimal tables the metadata snapshot exceeds max_mutation_size and fails with only a WARN.
## Commits, in review order
1-3. The three already-reviewed allocation fixes (21659/21660/21661), unchanged.
4. Remaining O(N) work: withAddedOrUpdated delta, ThreadLocalMeter geometric growth, validate.
5. Coalesced schema flush behind `schema_flush_coalescing_window` (default 1000ms, `0ms` = old behaviour). Drain unchanged. Crash-restart dtest.
6. Snapshot store failure: ERROR + metric; WARN at 75% of max_mutation_size.
7. tables_warn_threshold default 1000; docs; dtest heap cap fix (junit maxmemory overrode -Xmx8G).
## Measured (single node in-JVM, ManyTablesScalingTest, same build, only the setting differs)
| N | before (sync flush, commits 1-3 only) | this branch | ratio |
|---|---|---|---|
| 1,000 | 120.9 s | 29.9 s | 4.0x |
| 5,000 | 675.1 s | 204.5 s | 3.3x |
| 10,000 | 1680.3 s (28.0 min) | 579.4 s (9.7 min) | 2.9x |
Per-statement mean, first bucket -> last bucket: before 112 -> 234 ms at 10k; this branch 32 -> 95 ms at 10k.
The fixed floor is gone; a residual ~6 ms per 1,000 existing tables remains (TablesDiff full scans and the
delta loops in withAddedOrUpdated are O(N) in time though allocation-free). That is the follow-up.
## Durability note for commit 5
system_schema has durable_writes=true (mutations are in the commitlog before any flush), and under TCM the
cluster metadata log is the startup source of truth: system_schema is rewritten from it on replay. The unit
test suite has run with this flush disabled for years (build.xml sets cassandra.test.flush_local_schema_changes=false).
Drain flushes system keyspaces synchronously as before. Raised on dev@ ("[DISCUSS] Making the synchronous system_schema flush on every DDL async").
## Not in this PR
Snapshot chunking, a bulk (N statements per epoch) schema transformation, and the per-table memory footprint.
Follow-up tickets referenced from the JIRA.
## CI note
Commit 7 makes the dtest JVMs really use 8G (they have been running at 1G); pod memory for dtest jobs should be checked.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01FKHCBUx77dTsezikytmjwE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21664: Make schema changes cost O(1) in the number of tables and fail loudly at schema limits [cassandra]
Posted by "pmcfadin (via GitHub)" <gi...@apache.org>.
pmcfadin commented on PR #5128:
URL: https://github.com/apache/cassandra/pull/5128#issuecomment-6351912514
Rebased onto current cassandra-6.0 to clear a CHANGES.txt conflict; there are no code changes. The new head is 189413ad84, with a code tree identical to ef2862358e apart from the CHANGES.txt line, so a run on either is valid. Trunk branch: d1dbaa20ce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21664: Make schema changes cost O(1) in the number of tables and fail loudly at schema limits [cassandra]
Posted by "pmcfadin (via GitHub)" <gi...@apache.org>.
pmcfadin commented on PR #5128:
URL: https://github.com/apache/cassandra/pull/5128#issuecomment-6351699246
Jon, the hang is fixed, so this is ready for your re-run. The coalesced flush task no longer blocks the shared `nonPeriodicTasks` thread. It only starts the `system_schema` flushes now (ef2862358e, with a new Byteman regression test, `SchemaFlushCoalesceNonBlockingTest`).
CI after the fix showed no jvm-dtest hangs in three runs: 6.0 pre-commit #98, trunk pre-commit #57 and a trunk jvm-dtest-only run #56. Before the fix there were 6 hangs in three runs. The ci_summary and results_details files for both branches, and the failure triage, are on CASSANDRA-21664.
Could you re-run your pre-commit suite on ef2862358e and give your +1 if it's green? The trunk branch for the merge-up is pmcfadin/schema-at-scale-trunk (52bf89dcfe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21664: Make schema changes cost O(1) in the number of tables and fail loudly at schema limits [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on PR #5128:
URL: https://github.com/apache/cassandra/pull/5128#issuecomment-6061963909
Hi Patrick, thanks for the latest updates! The order-independent SnapshotsTest change fixed the fuzz SnapshotsTest failure I was seeing.
I ran the PR head (`e7ddf5c3e8`) through the full pre-commit suite on JDK 21 and found two things.
**1. `SinglePartitionReadCommandTest` (jvm-dtest) fails consistently.** It failed in every run of this branch (also after rebasing onto current `cassandra-6.0`) and never on `cassandra-6.0`. It shows up as either "Timeout occurred" or an `OutOfMemoryError: Java heap space` during cluster startup in the last two test methods.
I tracked it to "Coalesce system_schema flushes instead of flushing synchronously on every schema change". With the default `schema_flush_coalescing_window: 1000ms`, each in-jvm node shuts down less than a second after its last DDL, so the scheduled flush never runs. `Instance.shutdown` stops the flush executors before the scheduled executors, and the log shows:
```
WARN SchemaKeyspace.java:487 - Failed to flush system_schema tables
java.util.concurrent.RejectedExecutionException: MemtableFlushWriter has shut down
```
Each node therefore shuts down with non-empty `system_schema` memtables, each holding at least one 1 MiB slab with `heap_buffers`. In-jvm clusters stay in memory until the JVM exits (that is already true on 6.0, so this class runs close to the 1 GiB test heap even there). Heap histograms show about 7 MB more retained per node (19.2 MB of `byte[]` per node vs 12.1 MB on 6.0), which is enough to push the fork over.
To reproduce it with the same 1 GiB fork that CI uses: `ant testclasslist -Dtest.classlistprefix=distributed` with a class list holding only `SinglePartitionReadCommandTest`, on JDK 21 (ZGC). It passes 8/8 on 6.0 and fails on this branch. With `-Dtest.jvm.args="-Dcassandra.config.allow_system_properties=true -Dcassandra.settings.schema_flush_coalescing_window=0ms"` it passes again. `test-jvm-dtest-some` hides it, because it runs with `-Xmx8G`.
A suggestion: when a coalesced flush is pending at shutdown, cancel the scheduled task and do one last synchronous flush before the flush executors stop (`Instance.shutdown`, before `ColumnFamilyStore.shutdownExecutorsAndWait`, and on any non-drain shutdown path). That also fixes the flush running against stopped executors. I'd avoid setting the window to 0 for dtests, since that would turn the feature off under test.
**2. `cqlsh_tests/test_cqlsh.py::TestCqlsh::test_default_keyspaces_exist` fails, but only because of the base.** The branch is cut from a `cassandra-6.0` that predates CASSANDRA-21720 (`b86d17c92b`, "Do not expose system_metrics virtual keyspace by default"), so `DESCRIBE KEYSPACES` still lists `system_metrics`, while the current dtest (`ba9ea01a`, "Align test_cqlsh.py with CASSANDRA-21720") expects it hidden. A rebase onto current `cassandra-6.0` should clear it; nothing in your series needs to change for this one.
Happy to help test a fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21664: Make schema changes cost O(1) in the number of tables and fail loudly at schema limits [cassandra]
Posted by "pmcfadin (via GitHub)" <gi...@apache.org>.
pmcfadin commented on PR #5128:
URL: https://github.com/apache/cassandra/pull/5128#issuecomment-6326850798
Thanks Jon, great diagnosis. I implemented your suggestion in 6e93f64e19 (c0cf821f0e before the rebase). `SchemaKeyspace.shutdownCoalescedFlush()` cancels any pending coalesced flush and does one final synchronous flush. It is called from `StorageService.drain` and from in-jvm `Instance.shutdown` before the flush executors stop. `SchemaFlushCoalesceShutdownTest` fails without the fix. `SinglePartitionReadCommandTest` now passes locally with the 1 GiB test heap.
The branch is rebased onto current cassandra-6.0, which clears the cqlsh `test_default_keyspaces_exist` failure (CASSANDRA-21720), and I added CHANGES.txt/NEWS.txt entries.
Could you re-run your pre-commit suite on a40069ba77 and give your +1 if it's green?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21664: Make schema changes cost O(1) in the number of tables and fail loudly at schema limits [cassandra]
Posted by "pmcfadin (via GitHub)" <gi...@apache.org>.
pmcfadin commented on PR #5128:
URL: https://github.com/apache/cassandra/pull/5128#issuecomment-6345474803
Jon, please hold off on the re-run for now. CI on this branch is showing intermittent jvm-dtest hangs that look like they come from the patch.
The hangs are 15-minute class timeouts in ForceSnapshotTest, ReadRepairCollectionQueriesTest, ReadRepairInQueriesTest, TransientRangeMovement2Test and ShortReadProtectionTest, each on a single JDK. There were 6 across three runs of this patch, and none in a control run of the same branch with `schema_flush_coalescing_window` defaulted to 0ms, or in 8 unrelated builds.
The likely cause: the coalesced flush runs a blocking `system_schema` flush on the single-threaded `ScheduledExecutors.nonPeriodicTasks`, the same executor that dropping a table waits on to drain (`LifecycleTransaction.waitForDeletions`) and that TCM uses for commit retries. Before this patch that flush never ran on `nonPeriodicTasks`. I'm changing the coalesced flush so it no longer blocks that executor, and I'll ping here once CI is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org