[jira] [Created] (CASSANDRA-20141) Unresponsive node after ingesting large amounts of vectors

["Robert Knutsson (Jira)" <ji...@apache.org>]

Robert Knutsson created CASSANDRA-20141:
-------------------------------------------

             Summary: Unresponsive node after ingesting large amounts of vectors
                 Key: CASSANDRA-20141
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-20141
             Project: Apache Cassandra
          Issue Type: Bug
            Reporter: Robert Knutsson


{*}Background{*}:

We have a Cassandra 5.0.2 cluster running on java 17, we've tried with everything from 3 to 23 nodes (running in AWS on r7i.4xlarge instances)

We have a table with an id column of type TEXT and another column of type VECTOR <FLOAT, 256>.

On that table we also have an SAI index on the VECTOR column with the options \{ 'similarity_function': 'EUCLIDEAN' }

*When:*

When we ingest large amounts of embeddings (~200 million) we notice each and every time that before all embeddings are saved a node becomes unresponsive (after >20 million are ingested) and no other node is unable to rejoin the cluster.

If the index is removed before we ingest the data, everything is able to be properly persisted, but once the index is added (and created successfully) the same thing happens again once we continue writing more embeddings to the cluster

*What:*

We saw the following stacktrace in our logs:
{noformat}
java.lang.NullPointerException: Cannot invoke "java.lang.Boolean.booleanValue()" because "res" is null
    at org.apache.cassandra.utils.memory.MemtableCleanerThread$Clean.apply(MemtableCleanerThread.java:97)
    at org.apache.cassandra.utils.concurrent.ListenerList$CallbackBiConsumerListener.run(ListenerList.java:244)
    at org.apache.cassandra.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:140)
    at org.apache.cassandra.utils.concurrent.ListenerList.safeExecute(ListenerList.java:166)
    at org.apache.cassandra.utils.concurrent.ListenerList.notifyListener(ListenerList.java:157)
    at org.apache.cassandra.utils.concurrent.ListenerList$CallbackBiConsumerListener.notifySelf(ListenerList.java:250)
    at org.apache.cassandra.utils.concurrent.ListenerList.lambda$notifyExclusive$0(ListenerList.java:124)
    at org.apache.cassandra.utils.concurrent.IntrusiveStack.forEach(IntrusiveStack.java:195)
    at org.apache.cassandra.utils.concurrent.ListenerList.notifyExclusive(ListenerList.java:124)
    at org.apache.cassandra.utils.concurrent.ListenerList.notify(ListenerList.java:96)
    at org.apache.cassandra.utils.concurrent.AsyncFuture.trySet(AsyncFuture.java:104)
    at org.apache.cassandra.utils.concurrent.AbstractFuture.tryFailure(AbstractFuture.java:148)
    at org.apache.cassandra.utils.concurrent.AsyncPromise.tryFailure(AsyncPromise.java:139)
    at org.apache.cassandra.db.memtable.AbstractAllocatorMemtable.lambda$flushLargestMemtable$0(AbstractAllocatorMemtable.java:306)
    at org.apache.cassandra.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:140)
    at org.apache.cassandra.utils.concurrent.ListenerList.safeExecute(ListenerList.java:166)
    at org.apache.cassandra.utils.concurrent.ListenerList.notifyListener(ListenerList.java:157)
    at org.apache.cassandra.utils.concurrent.ListenerList$RunnableWithExecutor.notifySelf(ListenerList.java:345)
    at org.apache.cassandra.utils.concurrent.ListenerList.lambda$notifyExclusive$0(ListenerList.java:124)
    at org.apache.cassandra.utils.concurrent.IntrusiveStack.forEach(IntrusiveStack.java:195)
    at org.apache.cassandra.utils.concurrent.ListenerList.notifyExclusive(ListenerList.java:124)
    at org.apache.cassandra.utils.concurrent.ListenerList.notify(ListenerList.java:96)
    at org.apache.cassandra.utils.concurrent.AsyncFuture.trySet(AsyncFuture.java:104)
    at org.apache.cassandra.utils.concurrent.AbstractFuture.tryFailure(AbstractFuture.java:148)
    at org.apache.cassandra.concurrent.FutureTask.tryFailure(FutureTask.java:87)
    at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:75)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:840)
{noformat}
This leads me to believe the above NPE happens once the Memtables are to be cleaned (persisted as SSTables?) perhaps?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org