You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Hu Xi <hu...@hotmail.com> on 2017/11/17 06:08:14 UTC
答复: kafka.admin.TopicCommand Failing
Increasing `zookeeper.connection.timeout.ms` to a relatively larger value might be a help. Besides, you could check GC log to see if the STW pauses expired the Zk sessions.
________________________________
发件人: Abhimanyu Nagrath <ab...@gmail.com>
发送时间: 2017年11月17日 13:51
收件人: users@kafka.apache.org; user@zookeeper.apache.org
主题: Re: kafka.admin.TopicCommand Failing
One more thing was checking my Kafka-server.log its fill with the warning
Attempting to send response via channel for which there is no open
connection, connection id 2 (Kafka.network.Processor)
IS this the reason for the above issue? How to resolve this. Need help
production is breaking.
Regards,
Abhimanyu
On Thu, Nov 16, 2017 at 5:08 PM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:
> Hi, I am using a single node Kafka V 0.10.2 (16 GB RAM, 8 cores) and a
> single node zookeeper V 3.4.9 (4 GB RAM, 1 core ). I am having 64 consumer
> groups and 500 topics each having 250 partitions. I am able to execute the
> commands which require only Kafka broker and its running fine
> ex.
>
> > ./kafka-consumer-groups.sh --bootstrap-server localhost:9092
> > --describe --group <topic>
>
> But when I execute the admin command like create topic, alter topic For
> example
>
> > ./kafka-topics.sh --create --zookeeper <zookeeper>:2181
> > --replication-factor 1 --partitions 1 --topic <topic>
>
> Following exception is being displayed:
>
>
>
> > Error while executing topic command : replication factor: 1 larger
> > than available brokers: 0 [2017-11-16 11:22:13,592] ERROR
> > org.apache.kafka.common.errors.InvalidReplicationFactorException:
> > replication factor: 1 larger than available brokers: 0
> > (kafka.admin.TopicCommand$)
>
> I checked my broker is up. In server.log following warnings are there
>
> [2017-11-16 11:14:26,959] WARN Client session timed out, have not
> heard from server in 15843ms for sessionid 0x15aa7f586e1c061
> (org.apache.zookeeper.ClientCnxn)
> [2017-11-16 11:14:28,795] WARN Unable to reconnect to ZooKeeper
> service, session 0x15aa7f586e1c061 has expired (org.apache.zookeeper.
> ClientCnxn)
> [2017-11-16 11:21:46,055] WARN Unable to reconnect to ZooKeeper
> service, session 0x15aa7f586e1c067 has expired (org.apache.zookeeper.
> ClientCnxn)
>
> Below mentioned is my Kafka server configuration :
>
> broker.id=1
> delete.topic.enable=true
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/kafka/data/logs
> num.partitions=1
> log.segment.bytes=1073741824
> log.retention.check.interval.ms=300000
> zookeeper.connect=<zookeeperIP>:2181
> zookeeper.connection.timeout.ms=6000
>
> Zookeeper Configuration is :
>
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=10
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=5
> # the directory where the snapshot is stored.
> # do not use /tmp for storage, /tmp here is just
> # example sakes.
> dataDir=/zookeeper/data
> # the port at which the clients will connect
> clientPort=2181
> # the maximum number of client connections.
> # increase this if you need to handle more clients
> #maxClientCnxns=60
> autopurge.snapRetainCount=20
> # Purge task interval in hours
> # Set to "0" to disable auto purge feature
> autopurge.purgeInterval=48
>
> I am not able to figure out which configuration to tune. What I am missing
> .Any help will be appreciated.
>
>
>
>
> Regards,
> Abhimanyu
>
Re: 答复: kafka.admin.TopicCommand Failing
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
I am absolutely new to these technologies . Can you assist me with the
below mentioned queries :
1. How to decide the value of zookeeper.connection.timeout.ms ?
2. How to check GC log to see if the STW pauses expired the Zk sessions
3. How to tune GC .
Regards,
Abhimanyu
On Fri, Nov 17, 2017 at 11:38 AM, Hu Xi <hu...@hotmail.com> wrote:
> Increasing `zookeeper.connection.timeout.ms` to a relatively larger value
> might be a help. Besides, you could check GC log to see if the STW pauses
> expired the Zk sessions.
>
>
> ________________________________
> 发件人: Abhimanyu Nagrath <ab...@gmail.com>
> 发送时间: 2017年11月17日 13:51
> 收件人: users@kafka.apache.org; user@zookeeper.apache.org
> 主题: Re: kafka.admin.TopicCommand Failing
>
> One more thing was checking my Kafka-server.log its fill with the warning
>
> Attempting to send response via channel for which there is no open
> connection, connection id 2 (Kafka.network.Processor)
>
> IS this the reason for the above issue? How to resolve this. Need help
> production is breaking.
>
>
> Regards,
> Abhimanyu
>
> On Thu, Nov 16, 2017 at 5:08 PM, Abhimanyu Nagrath <
> abhimanyunagrath@gmail.com> wrote:
>
> > Hi, I am using a single node Kafka V 0.10.2 (16 GB RAM, 8 cores) and a
> > single node zookeeper V 3.4.9 (4 GB RAM, 1 core ). I am having 64
> consumer
> > groups and 500 topics each having 250 partitions. I am able to execute
> the
> > commands which require only Kafka broker and its running fine
> > ex.
> >
> > > ./kafka-consumer-groups.sh --bootstrap-server localhost:9092
> > > --describe --group <topic>
> >
> > But when I execute the admin command like create topic, alter topic For
> > example
> >
> > > ./kafka-topics.sh --create --zookeeper <zookeeper>:2181
> > > --replication-factor 1 --partitions 1 --topic <topic>
> >
> > Following exception is being displayed:
> >
> >
> >
> > > Error while executing topic command : replication factor: 1 larger
> > > than available brokers: 0 [2017-11-16 11:22:13,592] ERROR
> > > org.apache.kafka.common.errors.InvalidReplicationFactorException:
> > > replication factor: 1 larger than available brokers: 0
> > > (kafka.admin.TopicCommand$)
> >
> > I checked my broker is up. In server.log following warnings are there
> >
> > [2017-11-16 11:14:26,959] WARN Client session timed out, have not
> > heard from server in 15843ms for sessionid 0x15aa7f586e1c061
> > (org.apache.zookeeper.ClientCnxn)
> > [2017-11-16 11:14:28,795] WARN Unable to reconnect to ZooKeeper
> > service, session 0x15aa7f586e1c061 has expired (org.apache.zookeeper.
> > ClientCnxn)
> > [2017-11-16 11:21:46,055] WARN Unable to reconnect to ZooKeeper
> > service, session 0x15aa7f586e1c067 has expired (org.apache.zookeeper.
> > ClientCnxn)
> >
> > Below mentioned is my Kafka server configuration :
> >
> > broker.id=1
> > delete.topic.enable=true
> > num.network.threads=3
> > num.io.threads=8
> > socket.send.buffer.bytes=102400
> > socket.receive.buffer.bytes=102400
> > socket.request.max.bytes=104857600
> > log.dirs=/kafka/data/logs
> > num.partitions=1
> > log.segment.bytes=1073741824
> > log.retention.check.interval.ms=300000
> > zookeeper.connect=<zookeeperIP>:2181
> > zookeeper.connection.timeout.ms=6000
> >
> > Zookeeper Configuration is :
> >
> > # The number of milliseconds of each tick
> > tickTime=2000
> > # The number of ticks that the initial
> > # synchronization phase can take
> > initLimit=10
> > # The number of ticks that can pass between
> > # sending a request and getting an acknowledgement
> > syncLimit=5
> > # the directory where the snapshot is stored.
> > # do not use /tmp for storage, /tmp here is just
> > # example sakes.
> > dataDir=/zookeeper/data
> > # the port at which the clients will connect
> > clientPort=2181
> > # the maximum number of client connections.
> > # increase this if you need to handle more clients
> > #maxClientCnxns=60
> > autopurge.snapRetainCount=20
> > # Purge task interval in hours
> > # Set to "0" to disable auto purge feature
> > autopurge.purgeInterval=48
> >
> > I am not able to figure out which configuration to tune. What I am
> missing
> > .Any help will be appreciated.
> >
> >
> >
> >
> > Regards,
> > Abhimanyu
> >
>