You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Guozhang Wang (JIRA)" <ji...@apache.org> on 2017/05/08 17:41:05 UTC
[jira] [Commented] (KAFKA-5070)
org.apache.kafka.streams.errors.LockException: task [0_18] Failed to lock
the state directory: /opt/rocksdb/pulse10/0_18
[ https://issues.apache.org/jira/browse/KAFKA-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001178#comment-16001178 ]
Guozhang Wang commented on KAFKA-5070:
--------------------------------------
[~dnagarajan] [~shoyebpathan] The WARN entry in your stack trace does not necessarily indicate an error but just a transient state where when multiple threads on the same JVM are rebalancing, some thread may be trying to grab local file locks while others have not yet release them. It may looks a bit scary that we does print the exception stack trace from the WARN entry (we may consider fixing it to not include the exception in the log entry, [~mjsax]?). So the question is:
1. does this WARN log entry keep showing up and never going away?
2. when it shows up, does the Streams app stops progressing and never resumes anymore?
> org.apache.kafka.streams.errors.LockException: task [0_18] Failed to lock the state directory: /opt/rocksdb/pulse10/0_18
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-5070
> URL: https://issues.apache.org/jira/browse/KAFKA-5070
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.10.2.0
> Environment: Linux Version
> Reporter: Dhana
> Attachments: RocksDB_LockStateDirec.7z
>
>
> Notes: we run two instance of consumer in two difference machines/nodes.
> we have 400 partitions. 200 stream threads/consumer, with 2 consumer.
> We perform HA test(on rebalance - shutdown of one of the consumer/broker), we see this happening
> Error:
> 2017-04-05 11:36:09.352 WARN StreamThread:1184 StreamThread-66 - Could not create task 0_115. Will retry.
> org.apache.kafka.streams.errors.LockException: task [0_115] Failed to lock the state directory: /opt/rocksdb/pulse10/0_115
> at org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:102)
> at org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:73)
> at org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:108)
> at org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
> at org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
> at org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
> at org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
> at org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
> at org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
> at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
> at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
> at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)