You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by AjeetSingh <aj...@hotmail.com> on 2016/12/02 05:08:46 UTC
zookeeper discnnected every minutes when i am getting bulk data
Hi Folks,
I am facing some challenging issue in my cluster and that is related with
zookeeper request timeout.
A:- 3 Kafka cluster with zookeeper
B:- Zookeeper Veriosn is zookeeper-3.4.6
C:- Java Version java version "1.7.0_111"
OpenJDK Runtime Environment (IcedTea 2.6.7) (7u111-2.6.7-0ubuntu0.14.04.3)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)
D:- Kafka Version kafka_2.10-0.10.0.1
When i am running the zookeeper and getting timeout in every minutes, please
folle the below error.
[2016-12-02 09:46:30,869] INFO Client session timed out, have not heard from
server in 4002ms for sessionid 0x158b9f7c7200000, closing socket connection
and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-12-02 09:46:30,971] INFO zookeeper state changed (Disconnected)
(org.I0Itec.zkclient.ZkClient)
[2016-12-02 09:46:31,922] INFO Opening socket connection to server
10.0.2.3/10.0.2.3:2181. Will not attempt to authenticate using SASL (unknown
error) (org.apache.zookeeper.ClientCnxn)
[2016-12-02 09:46:31,924] INFO Socket connection established to
10.0.2.3/10.0.2.3:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-12-02 09:46:31,926] INFO Session establishment complete on server
10.0.2.3/10.0.2.3:2181, sessionid = 0x158b9f7c7200000, negotiated timeout =
6000 (org.apache.zookeeper.ClientCnxn)
[2016-12-02 09:46:31,926] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2016-12-02 09:50:15,163] INFO [Group Metadata Manager on Broker 2]: Removed
0 expired offsets in 0 milliseconds.
(kafka.coordinator.GroupMetadataManager)
<http://zookeeper-user.578899.n2.nabble.com/file/n7582817/zookeeper.png>
Please help me and resolve this problem.
Cheers,
Ajeet S
--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/zookeeper-discnnected-every-minutes-when-i-am-getting-bulk-data-tp7582817.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: zookeeper discnnected every minutes when i am getting bulk data
Posted by Rakesh Radhakrishnan <ra...@apache.org>.
Like Flavio mentioned in previous mail, could you please check ZooKeeper
server logs. IMHO, that will help to understand the status of the cluster
and we may get hints to identify the problem.
Thanks for detailing your architecture. Probably, will come back to this
after analyzing ZK server logs.
-Rakesh-
On Fri, Dec 2, 2016 at 3:45 PM, AjeetSingh <aj...@hotmail.com>
wrote:
> Hi,
> Please follow the below Arch for my setup.
>
>
> Logstash (Shipper)+Kafka/Zookeeper Cluster 3 nodes+Logstash
> (Indexer)+LB+ElasticSearch Cluster(8 nodes)+LB+KIbana
>
> This is my Arch for current setup but that would be scale in future but
> before going to scale need to resolve this issue.
>
> Please help me.
>
>
>
> Cheers,
> Ajeet S
>
>
>
>
>
>
> -----
> Cheers,
> Ajeet S
>
> --
> View this message in context: http://zookeeper-user.578899.
> n2.nabble.com/zookeeper-discnnected-every-minutes-
> when-i-am-getting-bulk-data-tp7582817p7582822.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
>
Re: zookeeper discnnected every minutes when i am getting bulk data
Posted by AjeetSingh <aj...@hotmail.com>.
Hi,
Please follow the below Arch for my setup.
Logstash (Shipper)+Kafka/Zookeeper Cluster 3 nodes+Logstash
(Indexer)+LB+ElasticSearch Cluster(8 nodes)+LB+KIbana
This is my Arch for current setup but that would be scale in future but
before going to scale need to resolve this issue.
Please help me.
Cheers,
Ajeet S
-----
Cheers,
Ajeet S
--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/zookeeper-discnnected-every-minutes-when-i-am-getting-bulk-data-tp7582817p7582822.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: zookeeper discnnected every minutes when i am getting bulk data
Posted by Flavio Junqueira <fp...@apache.org>.
If I understand this right, it is Kafka that is using ZooKeeper, not your application directly. The log excerpt you sent is from a Kafka broker.
You may want to inspect the server logs to determine if the ensemble is healthy.
-Flavio
> On 02 Dec 2016, at 06:41, AjeetSingh <aj...@hotmail.com> wrote:
>
> Hi Rakesh,
>
> I am using 3 Kafka/Zookeepers nodes and that is running cluster mode and i
> am getting 100 GB data per day in one topics and i have ELK cluster.
>
> So i am not getting zookeepers disconnected in each mins and getting lag
> when zookeepers disconnected and how can i see the jute.maxbuffer and
> define the exact value.
>
> Please help me.
>
> Cheers,
> Ajeet S
>
>
>
>
>
>
> -----
> Cheers,
> Ajeet S
>
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/zookeeper-discnnected-every-minutes-when-i-am-getting-bulk-data-tp7582817p7582820.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: zookeeper discnnected every minutes when i am getting bulk data
Posted by AjeetSingh <aj...@hotmail.com>.
Hi Rakesh,
I am using 3 Kafka/Zookeepers nodes and that is running cluster mode and i
am getting 100 GB data per day in one topics and i have ELK cluster.
So i am not getting zookeepers disconnected in each mins and getting lag
when zookeepers disconnected and how can i see the jute.maxbuffer and
define the exact value.
Please help me.
Cheers,
Ajeet S
-----
Cheers,
Ajeet S
--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/zookeeper-discnnected-every-minutes-when-i-am-getting-bulk-data-tp7582817p7582820.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: zookeeper discnnected every minutes when i am getting bulk data
Posted by Rakesh Radhakrishnan <ra...@apache.org>.
Hi Ajeet,
Are you talking about putting more data bytes(bulk data) into a single
zNode?
I hope you know that, the data size (or packet size) is controlled through
configuration property - Java system property: jute.maxbuffer
Reference: https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html
ZooKeeper will throw exception if the data size cross this value. Could you
check ZooKeeper server logs to see the chances of exceptions and due to
that the client requests are failing.
Thanks,
Rakesh
On Fri, Dec 2, 2016 at 10:38 AM, AjeetSingh <aj...@hotmail.com>
wrote:
> Hi Folks,
>
> I am facing some challenging issue in my cluster and that is related with
> zookeeper request timeout.
>
> A:- 3 Kafka cluster with zookeeper
> B:- Zookeeper Veriosn is zookeeper-3.4.6
> C:- Java Version java version "1.7.0_111"
> OpenJDK Runtime Environment (IcedTea 2.6.7) (7u111-2.6.7-0ubuntu0.14.04.3)
> OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)
> D:- Kafka Version kafka_2.10-0.10.0.1
>
> When i am running the zookeeper and getting timeout in every minutes,
> please
> folle the below error.
>
> [2016-12-02 09:46:30,869] INFO Client session timed out, have not heard
> from
> server in 4002ms for sessionid 0x158b9f7c7200000, closing socket connection
> and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2016-12-02 09:46:30,971] INFO zookeeper state changed (Disconnected)
> (org.I0Itec.zkclient.ZkClient)
> [2016-12-02 09:46:31,922] INFO Opening socket connection to server
> 10.0.2.3/10.0.2.3:2181. Will not attempt to authenticate using SASL
> (unknown
> error) (org.apache.zookeeper.ClientCnxn)
> [2016-12-02 09:46:31,924] INFO Socket connection established to
> 10.0.2.3/10.0.2.3:2181, initiating session (org.apache.zookeeper.
> ClientCnxn)
> [2016-12-02 09:46:31,926] INFO Session establishment complete on server
> 10.0.2.3/10.0.2.3:2181, sessionid = 0x158b9f7c7200000, negotiated timeout
> =
> 6000 (org.apache.zookeeper.ClientCnxn)
> [2016-12-02 09:46:31,926] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2016-12-02 09:50:15,163] INFO [Group Metadata Manager on Broker 2]:
> Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.GroupMetadataManager)
> <http://zookeeper-user.578899.n2.nabble.com/file/n7582817/zookeeper.png>
>
> Please help me and resolve this problem.
>
>
> Cheers,
> Ajeet S
>
>
>
>
>
>
> --
> View this message in context: http://zookeeper-user.578899.
> n2.nabble.com/zookeeper-discnnected-every-minutes-
> when-i-am-getting-bulk-data-tp7582817.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
>