You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefan Miklosovic (Jira)" <ji...@apache.org> on 2024/02/23 15:43:00 UTC
[jira] [Comment Edited] (CASSANDRA-19427) Fix concurrent access of ClientWarn causing AIOBE for SELECT WHERE IN queries with multiple coordinator-local partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-19427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820096#comment-17820096 ]
Stefan Miklosovic edited comment on CASSANDRA-19427 at 2/23/24 3:42 PM:
------------------------------------------------------------------------
Would be great if we covered all branches this occurs in. It seems to me like the change will be rather small so merging it from 3.11 (at least) up to trunk should not be a lot of work.
was (Author: smiklosovic):
Would be great if we covered all branches this occurs in. It seems to me like the change will rather small so merging it from 3.11 (at least) up to trunk should not be a lot of work.
> Fix concurrent access of ClientWarn causing AIOBE for SELECT WHERE IN queries with multiple coordinator-local partitions
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-19427
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19427
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Legacy/Local Write-Read Paths
> Reporter: Abe Ratnofsky
> Assignee: Abe Ratnofsky
> Priority: Normal
> Fix For: 3.11.x, 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> On one of our clusters, we noticed rare but periodic ArrayIndexOutOfBoundsExceptions:
>
> {code:java}
> message="Uncaught exception on thread Thread[ReadStage-3,5,main]"
> exception="java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2579)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.ArrayIndexOutOfBoundsException"{code}
>
>
> The error was in a Runnable, so the stacktrace didn't directly indicate where the error was coming from. We enabled JFR to log the underlying exception that was thrown:
>
> {code:java}
> message="Uncaught exception on thread Thread[ReadStage-2,5,main]" exception="java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 0
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2579)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 0
> at java.base/java.util.ArrayList.add(ArrayList.java:487)
> at java.base/java.util.ArrayList.add(ArrayList.java:499)
> at org.apache.cassandra.service.ClientWarn$State.add(ClientWarn.java:84)
> at org.apache.cassandra.service.ClientWarn$State.access$000(ClientWarn.java:77)
> at org.apache.cassandra.service.ClientWarn.warn(ClientWarn.java:51)
> at org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:596)
> at org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:95)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2260)
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2575)
> ... 6 more"{code}
>
>
> An AIOBE on ArrayList.add(E) should only be possible when multiple threads attempt to call the method at the same time.
>
> This was seen while executing a SELECT WHERE IN query with multiple partition keys. This exception could happen when multiple local reads are dispatched by the coordinator in org.apache.cassandra.service.reads.AbstractReadExecutor#makeRequests. In this case, multiple local reads exceed the tombstone warning threshold, so multiple tombstone warnings are added to the same ClientWarn.State reference. Currently, org.apache.cassandra.service.ClientWarn.State#warnings is an ArrayList, which isn't safe for concurrent modification, causing the AIOBE to be thrown.
>
> I have a patch available for this, and I'm preparing it now. The patch is simple - it just changes org.apache.cassandra.service.ClientWarn.State#warnings to a thread-safe CopyOnWriteArrayList. I also have a jvm-dtest that demonstrates the issue but doesn't need to be merged - it shows how a SELECT WHERE IN query with local reads that add client warnings can add to the same ClientWarn.State from different threads. I'll push that in a separate branch just for demonstration purposes.
>
> This appears to have been an issue since at least 3.11, that was the earliest release I checked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org