You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Divij Vaidya (Jira)" <ji...@apache.org> on 2023/09/21 10:06:00 UTC

[jira] [Comment Edited] (KAFKA-15481) Concurrency bug in RemoteIndexCache leads to IOException

    [ https://issues.apache.org/jira/browse/KAFKA-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767489#comment-17767489 ] 

Divij Vaidya edited comment on KAFKA-15481 at 9/21/23 10:05 AM:
----------------------------------------------------------------

> And in `RemoteIndexCache#remove`, we can now use:
{code:java}
`internalCache.asMap().computeIfPresent(key, () -> {// rename files and return null to remove the key})` {code}
-But this doesn't handle the case of invalidation? Caffeine will not call "RemoteIndexCache#remove()" during invalidation and the race condition will still occur when invalidation and get is happening at same time.-
Ah, you meant to say to use this in-addition to using evictionListener. Fair.


was (Author: divijvaidya):
> And in `RemoteIndexCache#remove`, we can now use:
{code:java}
`internalCache.asMap().computeIfPresent(key, () -> {// rename files and return null to remove the key})` {code}

But this doesn't handle the case of invalidation? Caffeine will not call "RemoteIndexCache#remove()" during invalidation and the race condition will still occur when invalidation and get is happening at same time.

> Concurrency bug in RemoteIndexCache leads to IOException
> --------------------------------------------------------
>
>                 Key: KAFKA-15481
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15481
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 3.6.0
>            Reporter: Divij Vaidya
>            Priority: Major
>             Fix For: 3.7.0
>
>
> RemoteIndexCache has a concurrency bug which leads to IOException while fetching data from remote tier.
> Below events in order of timeline -
> Thread 1 (cache thread): invalidates the entry, removalListener is invoked async, so the files have not been renamed to "deleted" suffix yet.
> Thread 2: (fetch thread): tries to find entry in cache, doesn't find it because it has been removed by 1, fetches the entry from S3, writes it to existing file (using replace existing)
> Thread 1: async removalListener is invoked, acquires a lock on old entry (which has been removed from cache), it renames the file to "deleted" and starts deleting it
> Thread 2: Tries to create in-memory/mmapped index, but doesn't find the file and hence, creates a new file of size 2GB in AbstractIndex constructor. JVM returns an error as it won't allow creation of 2GB random access file.
> *Potential Fix*
> Use EvictionListener instead of RemovalListener in Caffeine cache as per the documentation:
> {quote} When the operation must be performed synchronously with eviction, use {{Caffeine.evictionListener(RemovalListener)}} instead. This listener will only be notified when {{RemovalCause.wasEvicted()}} is true. For an explicit removal, {{Cache.asMap()}} offers compute methods that are performed atomically.{quote}
> This will ensure that removal from cache and marking the file with delete suffix is synchronously done, hence the above race condition will not occur.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)