You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by daniel brown <da...@gmail.com> on 2018/01/03 08:18:47 UTC
Kafka : To meny files open
Hello,
Currently Zookeeper and kafka do not work appropriately and the logs are
not streaming to logstash. The var-log partitions across the zookeeper
cluster are filled up to 100 percent. We are getting the following
messages when we investigated the logs and the services status:
1 . Zookeeper logs
2018-01-03 06:16:55,001 [myid:1] - WARN
[SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2018-01-03 06:23:08,943 [myid:1] - WARN
[RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id
3, my id = 1, error = java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2018-01-03 06:23:08,943 [myid:1] - WARN
[RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2018-01-03 06:23:08,943 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2181
:Follower@87] - Exception when following the leader java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.
**Consumer check output
kafka-console-consumer --bootstrap-server localhost:2181 --topic
RCCSWA_DHCP_1 --from-beginning
[2018-01-03 07:39:27,559] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
[2018-01-03 07:39:27,660] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
[2018-01-03 07:39:27,761] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
2. Kafka Logs
tail -f server.log.2018-01-02-13
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:326)
at kafka.network.Acceptor.run(SocketServer.scala:269)
at java.lang.Thread.run(Thread.java:748)
[2018-01-02 13:30:01,124] ERROR Error while accepting connection
(kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChan
Thanks.
Re: Kafka : To meny files open
Posted by Matt Andruff <ma...@gmail.com>.
You should check the kafka docs. You need to increase your open file
descriptors.
Here's a link that describes how to do
it. (Ignore that this is for hdp the ulimit part is what you need to pay
attend to.)
You can also check kafka docs too they say to change open file descriptors
to 100,000 I believe.
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/kerb-config-limits.html
On Jan 3, 2018 10:48 AM, "daniel brown" <da...@gmail.com> wrote:
Hello,
Currently Zookeeper and kafka do not work appropriately and the logs are
not streaming to logstash. The var-log partitions across the zookeeper
cluster are filled up to 100 percent. We are getting the following
messages when we investigated the logs and the services status:
1 . Zookeeper logs
2018-01-03 06:16:55,001 [myid:1] - WARN
[SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2018-01-03 06:23:08,943 [myid:1] - WARN
[RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id
3, my id = 1, error = java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(
QuorumCnxManager.java:795)
2018-01-03 06:23:08,943 [myid:1] - WARN
[RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2018-01-03 06:23:08,943 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2181
:Follower@87] - Exception when following the leader java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.
**Consumer check output
kafka-console-consumer --bootstrap-server localhost:2181 --topic
RCCSWA_DHCP_1 --from-beginning
[2018-01-03 07:39:27,559] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
[2018-01-03 07:39:27,660] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
[2018-01-03 07:39:27,761] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
2. Kafka Logs
tail -f server.log.2018-01-02-13
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:326)
at kafka.network.Acceptor.run(SocketServer.scala:269)
at java.lang.Thread.run(Thread.java:748)
[2018-01-02 13:30:01,124] ERROR Error while accepting connection
(kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChan
Thanks.
Re: Kafka : To meny files open
Posted by Matt Andruff <ma...@gmail.com>.
You should check the kafka docs. You need to increase your open file
descriptors.
Here's a link that describes how to do
it. (Ignore that this is for hdp the ulimit part is what you need to pay
attend to.)
You can also check kafka docs too they say to change open file descriptors
to 100,000 I believe.
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/kerb-config-limits.html
On Jan 3, 2018 10:48 AM, "daniel brown" <da...@gmail.com> wrote:
Hello,
Currently Zookeeper and kafka do not work appropriately and the logs are
not streaming to logstash. The var-log partitions across the zookeeper
cluster are filled up to 100 percent. We are getting the following
messages when we investigated the logs and the services status:
1 . Zookeeper logs
2018-01-03 06:16:55,001 [myid:1] - WARN
[SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2018-01-03 06:23:08,943 [myid:1] - WARN
[RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id
3, my id = 1, error = java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(
QuorumCnxManager.java:795)
2018-01-03 06:23:08,943 [myid:1] - WARN
[RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2018-01-03 06:23:08,943 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2181
:Follower@87] - Exception when following the leader java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.
**Consumer check output
kafka-console-consumer --bootstrap-server localhost:2181 --topic
RCCSWA_DHCP_1 --from-beginning
[2018-01-03 07:39:27,559] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
[2018-01-03 07:39:27,660] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
[2018-01-03 07:39:27,761] WARN Bootstrap broker localhost:2181 disconnected
(org.apache.kafka.clients.NetworkClient)
2. Kafka Logs
tail -f server.log.2018-01-02-13
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:326)
at kafka.network.Acceptor.run(SocketServer.scala:269)
at java.lang.Thread.run(Thread.java:748)
[2018-01-02 13:30:01,124] ERROR Error while accepting connection
(kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChan
Thanks.