[jira] [Comment Edited] (CASSANDRA-21684) Unbounded per-thread cache of read-ahead buffers can cause slow heap leak

["Jon Haddad (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-21684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115366#comment-18115366 ] 

Jon Haddad edited comment on CASSANDRA-21684 at 9/15/26 1:01 AM:
-----------------------------------------------------------------

Fyi, I've got a patch I'm about to merge that removes the pool completely.  It doesn't really help improve performance and it introduced an issue with direct io, so I'm removing it in CASSANDRA-21671.

If you want to remove it in 5.0 instead, feel free to adapt this patch and merge it instead, though you might also find in mine we're also not correctly using the chunk cache.  I was going to do mine against 6.0, but if we have a problem with 5.0 too I can target that instead.


was (Author: rustyrazorblade):
Fyi, I've got a patch I'm about to merge that removes the pool completely.  It doesn't really help improve performance and it introduced an issue with direct io, so I'm removing it in CASSANDRA-21671.

> Unbounded per-thread cache of read-ahead buffers can cause slow heap leak
> -------------------------------------------------------------------------
>
>                 Key: CASSANDRA-21684
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21684
>             Project: Apache Cassandra
>          Issue Type: Bug
>          Components: Local/Compaction
>            Reporter: Caleb Rackliffe
>            Assignee: Caleb Rackliffe
>            Priority: Normal
>             Fix For: 5.0.x, 6.0.x, 7.x
>
>
> {{ThreadLocalReadAheadBuffer}} caches one read-ahead Block per scanned file in a per-thread map (blockMap), keyed by file path, so a thread can reuse its buffer across repeated scans of the same SSTable. That map has no upper bound and entries are only removed when the same thread that populated one also happens to run the file's final close, which rarely lines up, since SSTables are typically scanned by several different pool threads (compaction, repair, reads) over their lifetime. The result: every long-lived thread accumulates a permanent stale entry for every distinct SSTable it has ever scanned, growing unbounded as compaction continuously creates new SSTable generations.
> {noformat}
> @Test
> public void releasingScanResourcesShouldRemoveThreadLocalBlockMapEntry() throws Exception
> {
>     SequentialWriterOption writerOption = SequentialWriterOption.newBuilder().finishOnClose(false).bufferSize(1 << 10).build();
>     CompressionParams params = CompressionParams.snappy(4096, 1.1);
>     FileSystems.newGlobalInMemoryFileSystem();
>     File f = new File("/scan_release_repro.db");
>     File offsets = new File("/scan_release_repro.offset");
>     File digest = new File("/scan_release_repro.digest");
>     long longsToWrite = 600;
>     CompressionMetadata metadata;
>     try (CompressedSequentialWriter writer = new CompressedSequentialWriter(f, offsets, digest, writerOption, params, new MetadataCollector(new ClusteringComparator())))
>     {
>         for (long i = 0; i < longsToWrite; i++)
>             writer.writeLong(i);
>         writer.sync();
>         metadata = writer.open(0);
>     }
>     DatabaseDescriptor.setCompressedReadAheadBufferSizeInKb(256);
>     ByteBuffer buffer = ByteBuffer.allocateDirect(metadata.chunkLength());
>     try (ChannelProxy channel = new ChannelProxy(f);
>          CompressedChunkReader reader = new CompressedChunkReader.Standard(channel, metadata, () -> 1.1);
>          metadata)
>     {
>         reader.forScan();
>         reader.readChunk(0, buffer);
>         Assert.assertTrue("expected a read-ahead Block to be present for this file on this thread after a scan read",
>                            threadLocalBlockMapContainsKey(channel.filePath()));
>         // Mirrors what BufferManagingRebufferer.closeReader() does at the end of every scan.
>         reader.releaseUnderlyingResources();
>         Assert.assertFalse("a stale Block entry for this file should not remain in this thread's read-ahead " +
>                             "blockMap once the scan has released its resources -- otherwise every distinct " +
>                             "file this thread ever scans leaks a permanent entry",
>                             threadLocalBlockMapContainsKey(channel.filePath()));
>     }
>     finally
>     {
>         FileUtils.clean(buffer);
>     }
> }
> @SuppressWarnings("unchecked")
> private static boolean threadLocalBlockMapContainsKey(String filePath) throws Exception
> {
>     Field field = ThreadLocalReadAheadBuffer.class.getDeclaredField("blockMap");
>     field.setAccessible(true);
>     FastThreadLocal<Map<String, ?>> blockMap = (FastThreadLocal<Map<String, ?>>) field.get(null);
>     return blockMap.get().containsKey(filePath);
> }
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org