You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Abhimanyu Nagrath <ab...@gmail.com> on 2017/01/01 17:09:07 UTC
Zookeeper using high CPU
I am new to zookeeper and I have a single node zookeeper setup on a
system having 2 cores and 16 GB RAM. The number of active connections
on zookeeper is near about 700.And it is taking 2 GB of RAM and 100%
CPU(on both the cores). I am unable to find out what is causing this
high CPU Utilization
My Zookeeper Config is:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/path
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this
is a non-production config
maxClientCnxns=0
and the zookeeper logs are:
[2016-12-30 08:14:47,057] INFO Client attempting to establish new
session at /10.179.63.152:43480
(org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:14:47,107] INFO Established session
0x1594ae64cde0188 with negotiated timeout 10000 for client
/xx.xxx.xx.xxx:43480 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:14:48,099] INFO Got user-level KeeperException when
processing sessionid:0x1594ae64cde0188 type:create cxid:0x2
zxid:0xa798c81b txntype:-1 reqpath:n/a Error Path:/consumers
Error:KeeperErrorCode = NodeExists for /consumers
(org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:14:48,100] INFO Got user-level KeeperException when
processing sessionid:0x1594ae64cde0188 type:create cxid:0x3
zxid:0xa798c81c txntype:-1 reqpath:n/a Error Path:/consumers/prod
Error:KeeperErrorCode = NodeExists for /consumers/prod
(org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:14:48,101] INFO Got user-level KeeperException when
processing sessionid:0x1594ae64cde0188 type:create cxid:0x4
zxid:0xa798c81d txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids
Error:KeeperErrorCode = NodeExists for /consumers/prod/ids
(org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:25,273] INFO Accepted socket connection from
/10.185.3.226:58898 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-12-30 08:15:25,284] INFO Client attempting to establish new
session at /xx.xxx.xx.xxx:58898
(org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:25,347] INFO Established session
0x1594ae64cde0189 with negotiated timeout 10000 for client
/xx.xxx.xx.xxx:58898 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:26,762] INFO Got user-level KeeperException when
processing sessionid:0x1594ae64cde0189 type:create cxid:0x2
zxid:0xa798cb43 txntype:-1 reqpath:n/a Error Path:/consumers
Error:KeeperErrorCode = NodeExists for /consumers
(org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:26,762] INFO Got user-level KeeperException when
processing sessionid:0x1594ae64cde0189 type:create cxid:0x3
zxid:0xa798cb44 txntype:-1 reqpath:n/a Error Path:/consumers/prod
Error:KeeperErrorCode = NodeExists for /consumers/prod
(org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:26,763] INFO Got user-level KeeperException when
processing sessionid:0x1594ae64cde0189 type:create cxid:0x4
zxid:0xa798cb45 txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids
Error:KeeperErrorCode = NodeExists for /consumers/prod/ids
(org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:56,786] INFO Accepted socket connection from
/xx.xxx.xx.xxx:43574
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-12-30 08:15:56,790] INFO Client attempting to establish new
session at /xx.xxx.xx.xxx:43574
(org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:56,843] INFO Established session
0x1594ae64cde018a with negotiated timeout 10000 for client
/10.185.178.18:43574 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:57,743] INFO Got user-level KeeperException when
processing sessionid:0x1594ae64cde018a type:create cxid:0x2
zxid:0xa798db9d txntype:-1 reqpath:n/a Error Path:/consumers
Error:KeeperErrorCode = NodeExists for /consumers
(org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:57,743] INFO Got user-level KeeperException when
processing sessionid:0x1594ae64cde018a type:create cxid:0x3
zxid:0xa798db9e txntype:-1 reqpath:n/a Error Path:/consumers/prod
Error:KeeperErrorCode = NodeExists for /consumers/prod
(org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:57,744] INFO Got user-level KeeperException when
processing sessionid:0x1594ae64cde018a type:create cxid:0x4
zxid:0xa798db9f txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids
Error:KeeperErrorCode = NodeExists for /consumers/prod/ids
(org.apache.zookeeper.server.PrepRequestProcessor)
Can someone help me identify the problem .
Regards,
Abhimanyu
Re: Zookeeper using high CPU
Posted by Chris Nauroth <cn...@apache.org>.
Hello Abhimanyu,
Were you expecting to see those 700 active connections, or is it possible
that something is causing unexpected traffic?
One possibility is that this could be a JVM garbage collection problem. If
it's spending a lot of time in garbage collection, then that would show up
as high CPU utilization. Enabling garbage collection logs would reveal
more information about that.
Chris Nauroth
On Sun, Jan 1, 2017 at 9:09 AM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:
> I am new to zookeeper and I have a single node zookeeper setup on a
> system having 2 cores and 16 GB RAM. The number of active connections
> on zookeeper is near about 700.And it is taking 2 GB of RAM and 100%
> CPU(on both the cores). I am unable to find out what is causing this
> high CPU Utilization
>
> My Zookeeper Config is:
>
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version
> 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> # the directory where the snapshot is stored.
> dataDir=/path
> # the port at which the clients will connect
> clientPort=2181
> # disable the per-ip limit on the number of connections since this
> is a non-production config
> maxClientCnxns=0
>
> and the zookeeper logs are:
>
> [2016-12-30 08:14:47,057] INFO Client attempting to establish new
> session at /10.179.63.152:43480
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-12-30 08:14:47,107] INFO Established session
> 0x1594ae64cde0188 with negotiated timeout 10000 for client
> /xx.xxx.xx.xxx:43480 (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-12-30 08:14:48,099] INFO Got user-level KeeperException when
> processing sessionid:0x1594ae64cde0188 type:create cxid:0x2
> zxid:0xa798c81b txntype:-1 reqpath:n/a Error Path:/consumers
> Error:KeeperErrorCode = NodeExists for /consumers
> (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-12-30 08:14:48,100] INFO Got user-level KeeperException when
> processing sessionid:0x1594ae64cde0188 type:create cxid:0x3
> zxid:0xa798c81c txntype:-1 reqpath:n/a Error Path:/consumers/prod
> Error:KeeperErrorCode = NodeExists for /consumers/prod
> (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-12-30 08:14:48,101] INFO Got user-level KeeperException when
> processing sessionid:0x1594ae64cde0188 type:create cxid:0x4
> zxid:0xa798c81d txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids
> Error:KeeperErrorCode = NodeExists for /consumers/prod/ids
> (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-12-30 08:15:25,273] INFO Accepted socket connection from
> /10.185.3.226:58898 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-12-30 08:15:25,284] INFO Client attempting to establish new
> session at /xx.xxx.xx.xxx:58898
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-12-30 08:15:25,347] INFO Established session
> 0x1594ae64cde0189 with negotiated timeout 10000 for client
> /xx.xxx.xx.xxx:58898 (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-12-30 08:15:26,762] INFO Got user-level KeeperException when
> processing sessionid:0x1594ae64cde0189 type:create cxid:0x2
> zxid:0xa798cb43 txntype:-1 reqpath:n/a Error Path:/consumers
> Error:KeeperErrorCode = NodeExists for /consumers
> (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-12-30 08:15:26,762] INFO Got user-level KeeperException when
> processing sessionid:0x1594ae64cde0189 type:create cxid:0x3
> zxid:0xa798cb44 txntype:-1 reqpath:n/a Error Path:/consumers/prod
> Error:KeeperErrorCode = NodeExists for /consumers/prod
> (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-12-30 08:15:26,763] INFO Got user-level KeeperException when
> processing sessionid:0x1594ae64cde0189 type:create cxid:0x4
> zxid:0xa798cb45 txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids
> Error:KeeperErrorCode = NodeExists for /consumers/prod/ids
> (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-12-30 08:15:56,786] INFO Accepted socket connection from
> /xx.xxx.xx.xxx:43574
> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-12-30 08:15:56,790] INFO Client attempting to establish new
> session at /xx.xxx.xx.xxx:43574
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-12-30 08:15:56,843] INFO Established session
> 0x1594ae64cde018a with negotiated timeout 10000 for client
> /10.185.178.18:43574 (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-12-30 08:15:57,743] INFO Got user-level KeeperException when
> processing sessionid:0x1594ae64cde018a type:create cxid:0x2
> zxid:0xa798db9d txntype:-1 reqpath:n/a Error Path:/consumers
> Error:KeeperErrorCode = NodeExists for /consumers
> (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-12-30 08:15:57,743] INFO Got user-level KeeperException when
> processing sessionid:0x1594ae64cde018a type:create cxid:0x3
> zxid:0xa798db9e txntype:-1 reqpath:n/a Error Path:/consumers/prod
> Error:KeeperErrorCode = NodeExists for /consumers/prod
> (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-12-30 08:15:57,744] INFO Got user-level KeeperException when
> processing sessionid:0x1594ae64cde018a type:create cxid:0x4
> zxid:0xa798db9f txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids
> Error:KeeperErrorCode = NodeExists for /consumers/prod/ids
> (org.apache.zookeeper.server.PrepRequestProcessor)
>
> Can someone help me identify the problem .
>
>
> Regards,
> Abhimanyu
>