You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Guozhang Wang (Jira)" <ji...@apache.org> on 2022/02/05 02:55:00 UTC
[jira] [Commented] (KAFKA-13346) Kafka Streams fails due to RocksDB Locks Not Available Exception
[ https://issues.apache.org/jira/browse/KAFKA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487405#comment-17487405 ]
Guozhang Wang commented on KAFKA-13346:
---------------------------------------
I'm closing this ticket for now since we have not heard back from Amit. Please feel free to re-open it if there are new updates.
> Kafka Streams fails due to RocksDB Locks Not Available Exception
> ----------------------------------------------------------------
>
> Key: KAFKA-13346
> URL: https://issues.apache.org/jira/browse/KAFKA-13346
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.6.0
> Reporter: Amit Gupta
> Priority: Major
>
> Hello,
> We are using Kafka Streams and we observe that some times on some of the hosts running streams application, Kafka streams instance fails with unexpected exception. We are running with 40 stream threads per host and 20 hosts in total.
> Can some one please help on what can be the root cause here?
>
> |org.apache.kafka.streams.errors.ProcessorStateException: Error opening store state-store at location .....
> at org.apache.kafka.streams.state.internals.RocksDBStore.openRocksDB(RocksDBStore.java:214) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:188) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:224) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:42) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$0(MeteredKeyValueStore.java:101) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:836) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:101) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.ProcessorStateManager.registerStateStores(ProcessorStateManager.java:199) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:76) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StandbyTask.initializeIfNeeded(StandbyTask.java:95) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:426) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:660) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) ~[kafka-streams-2.6.0.jar:?]
> Caused by: org.rocksdb.RocksDBException: lock : ...../0_468/rocksdb/state-store/LOCK: No locks available
> at org.rocksdb.RocksDB.open(Native Method) ~[rocksdbjni-5.18.3.jar:?]
> at org.rocksdb.RocksDB.open(RocksDB.java:286) ~[rocksdbjni-5.18.3.jar:?]
> at org.apache.kafka.streams.state.internals.RocksDBStore.openRocksDB(RocksDBStore.java:211) ~[kafka-streams-2.6.0.jar:?]
> ... 15 more
>
> Some times I also see this exception
> |
> |org.apache.kafka.streams.errors.ProcessorStateException: Error opening store state-store at location ...../0_433/rocksdb/state-store
> at org.apache.kafka.streams.state.internals.RocksDBStore.openRocksDB(RocksDBStore.java:214) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:188) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:224) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:42) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$0(MeteredKeyValueStore.java:101) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:836) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:101) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.ProcessorStateManager.registerStateStores(ProcessorStateManager.java:199) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:76) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StandbyTask.initializeIfNeeded(StandbyTask.java:95) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:426) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:660) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) ~[kafka-streams-2.6.0.jar:?]
> at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) ~[kafka-streams-2.6.0.jar:?]
> Caused by: org.rocksdb.RocksDBException: while open a file for lock: ...../0_433/rocksdb/state-store/LOCK: Too many open files
> at org.rocksdb.RocksDB.open(Native Method) ~[rocksdbjni-5.18.3.jar:?]
> at org.rocksdb.RocksDB.open(RocksDB.java:286) ~[rocksdbjni-5.18.3.jar:?]
> at org.apache.kafka.streams.state.internals.RocksDBStore.openRocksDB(RocksDBStore.java:211) ~[kafka-streams-2.6.0.jar:?]
> ... 15 more|
--
This message was sent by Atlassian Jira
(v8.20.1#820001)