You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Qi Xu <sh...@gmail.com> on 2015/09/12 02:53:27 UTC
Kafka cluster cannot start anymore after unexpected shutdown
Hi,
We're running the Trunk version of Kafka (for its SSL feature) and recently
I'm trying to enable the kafka manager with it.
After enabling that, I find out some machine's Kafka Server is dead.
Looking at the server.log, it has the following logs.
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:41,310] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:44,766] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:48,567] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:52,112] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:56,307] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
And then I tried to start Kafka on these dead machines, but find the errors
below:
[2015-09-12 00:43:44,878] WARN Found an corrupted index file,
/datadrive/kafka-logs/userlogs-139/00000000000000000000.index, deleting and
rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,883] WARN Found an corrupted index file,
/datadrive/kafka-logs/userlogs-166/00000000000000000000.index, deleting and
rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,888] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-114/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,895] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-15/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,900] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-178/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,906] WARN Found an corrupted index file,
/datadrive/kafka-logs/__consumer_offsets-4/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,912] WARN Found an corrupted index file,
/datadrive/kafka-logs/userlogs-147/00000000000000000000.index, deleting and
rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,917] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-95/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
Exception in thread "kafka-log-cleaner-thread-0"
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"
[2015-09-12 00:44:39,466] WARN Session 0x14f84cab5240046 for server
10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2015-09-12 00:44:45,169] ERROR from main-SendThread(10.1.130.12:2181)
(org.apache.zookeeper.ClientCnxn)
[2015-09-12 00:45:39,466] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:39,467] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:39,732] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:41,299] ERROR Uncaught exception in scheduled task
'highwatermark-checkpoint' (kafka.utils.KafkaScheduler)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:49,070] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:49,702] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:49,912] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:50,140] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
>From zookeeper, I can see that node appear in /brokers/ids, but disappear
when the error above happens.
Do you know what's the problem? Any hint is very appreciated.
Thanks,
Qi
Re: Kafka cluster cannot start anymore after unexpected shutdown
Posted by Qi Xu <sh...@gmail.com>.
And I tried to clean up the whole kafka-logs folder, and then starts the
kafka server again.
It will the following errors:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/share/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/share/kafka/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2015-09-12 00:58:32,203] WARN No meta.properties file under dir
/datadrive/kafka-logs/meta.properties
(kafka.server.BrokerMetadataCheckpoint)
Exception in thread "kafka-log-cleaner-thread-0"
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-33"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-51"
log4j:ERROR No output stream or file set for the appender named
[kafkaAppender].
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-56"
[2015-09-12 00:59:54,546] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-33': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:34,261] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-51': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:51,108] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-56': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:48,867] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:00:48,867] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:00:49,121] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:57,643] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:54,028] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-40': (org.apache.kafka.common.utils.Utils)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:53,656] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:53,656] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-58': (org.apache.kafka.common.utils.Utils)
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-46"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-61"
[2015-09-12 01:00:53,043] WARN Session 0x24f84cb22a90029 for server
10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2015-09-12 01:01:22,419] ERROR from main-SendThread(10.1.130.12:2181)
(org.apache.zookeeper.ClientCnxn)
[2015-09-12 01:01:22,051] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:01:23,489] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-46': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:01:21,313] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:01:24,438] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-61': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:01:18,075] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:02,539] ERROR Uncaught exception in scheduled task
'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:34,023] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:35,676] ERROR Uncaught exception in scheduled task
'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler)
And the broker does not appear in zookeeper.
Any idea?
Thanks,
Qi
On Fri, Sep 11, 2015 at 5:53 PM, Qi Xu <sh...@gmail.com> wrote:
> Hi,
> We're running the Trunk version of Kafka (for its SSL feature) and
> recently I'm trying to enable the kafka manager with it.
> After enabling that, I find out some machine's Kafka Server is dead.
> Looking at the server.log, it has the following logs.
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:41,310] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:44,766] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:48,567] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:52,112] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:56,307] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
>
>
> And then I tried to start Kafka on these dead machines, but find the
> errors below:
> [2015-09-12 00:43:44,878] WARN Found an corrupted index file,
> /datadrive/kafka-logs/userlogs-139/00000000000000000000.index, deleting and
> rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,883] WARN Found an corrupted index file,
> /datadrive/kafka-logs/userlogs-166/00000000000000000000.index, deleting and
> rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,888] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-114/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,895] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-15/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,900] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-178/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,906] WARN Found an corrupted index file,
> /datadrive/kafka-logs/__consumer_offsets-4/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,912] WARN Found an corrupted index file,
> /datadrive/kafka-logs/userlogs-147/00000000000000000000.index, deleting and
> rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,917] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-95/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> Exception in thread "kafka-log-cleaner-thread-0"
> java.lang.OutOfMemoryError: Java heap space
>
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"
> [2015-09-12 00:44:39,466] WARN Session 0x14f84cab5240046 for server
> 10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
> and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2015-09-12 00:44:45,169] ERROR from main-SendThread(10.1.130.12:2181)
> (org.apache.zookeeper.ClientCnxn)
> [2015-09-12 00:45:39,466] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:39,467] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:39,732] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:41,299] ERROR Uncaught exception in scheduled task
> 'highwatermark-checkpoint' (kafka.utils.KafkaScheduler)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:49,070] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:49,702] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:49,912] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:50,140] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
>
> From zookeeper, I can see that node appear in /brokers/ids, but disappear
> when the error above happens.
>
> Do you know what's the problem? Any hint is very appreciated.
>
> Thanks,
> Qi
>
>
Re: Kafka cluster cannot start anymore after unexpected shutdown
Posted by Qi Xu <sh...@gmail.com>.
And I tried to clean up the whole kafka-logs folder, and then starts the
kafka server again.
It will the following errors:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/share/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/share/kafka/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2015-09-12 00:58:32,203] WARN No meta.properties file under dir
/datadrive/kafka-logs/meta.properties
(kafka.server.BrokerMetadataCheckpoint)
Exception in thread "kafka-log-cleaner-thread-0"
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-33"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-51"
log4j:ERROR No output stream or file set for the appender named
[kafkaAppender].
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-56"
[2015-09-12 00:59:54,546] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-33': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:34,261] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-51': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:51,108] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-56': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:48,867] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:00:48,867] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:00:49,121] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:57,643] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:54,028] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-40': (org.apache.kafka.common.utils.Utils)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:53,656] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:53,656] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-58': (org.apache.kafka.common.utils.Utils)
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-46"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-61"
[2015-09-12 01:00:53,043] WARN Session 0x24f84cb22a90029 for server
10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2015-09-12 01:01:22,419] ERROR from main-SendThread(10.1.130.12:2181)
(org.apache.zookeeper.ClientCnxn)
[2015-09-12 01:01:22,051] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:01:23,489] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-46': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:01:21,313] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:01:24,438] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-61': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:01:18,075] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:02,539] ERROR Uncaught exception in scheduled task
'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:34,023] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:35,676] ERROR Uncaught exception in scheduled task
'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler)
And the broker does not appear in zookeeper.
Any idea?
Thanks,
Qi
On Fri, Sep 11, 2015 at 5:53 PM, Qi Xu <sh...@gmail.com> wrote:
> Hi,
> We're running the Trunk version of Kafka (for its SSL feature) and
> recently I'm trying to enable the kafka manager with it.
> After enabling that, I find out some machine's Kafka Server is dead.
> Looking at the server.log, it has the following logs.
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:41,310] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:44,766] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:48,567] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:52,112] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:56,307] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
>
>
> And then I tried to start Kafka on these dead machines, but find the
> errors below:
> [2015-09-12 00:43:44,878] WARN Found an corrupted index file,
> /datadrive/kafka-logs/userlogs-139/00000000000000000000.index, deleting and
> rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,883] WARN Found an corrupted index file,
> /datadrive/kafka-logs/userlogs-166/00000000000000000000.index, deleting and
> rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,888] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-114/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,895] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-15/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,900] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-178/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,906] WARN Found an corrupted index file,
> /datadrive/kafka-logs/__consumer_offsets-4/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,912] WARN Found an corrupted index file,
> /datadrive/kafka-logs/userlogs-147/00000000000000000000.index, deleting and
> rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,917] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-95/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> Exception in thread "kafka-log-cleaner-thread-0"
> java.lang.OutOfMemoryError: Java heap space
>
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"
> [2015-09-12 00:44:39,466] WARN Session 0x14f84cab5240046 for server
> 10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
> and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2015-09-12 00:44:45,169] ERROR from main-SendThread(10.1.130.12:2181)
> (org.apache.zookeeper.ClientCnxn)
> [2015-09-12 00:45:39,466] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:39,467] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:39,732] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:41,299] ERROR Uncaught exception in scheduled task
> 'highwatermark-checkpoint' (kafka.utils.KafkaScheduler)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:49,070] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:49,702] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:49,912] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:50,140] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
>
> From zookeeper, I can see that node appear in /brokers/ids, but disappear
> when the error above happens.
>
> Do you know what's the problem? Any hint is very appreciated.
>
> Thanks,
> Qi
>
>