You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by sourabh chaki <ch...@gmail.com> on 2013/03/05 15:48:11 UTC
Re: Get all the partion details from zookeeper for a topic
Hi Neha,
Thanks for your reply.
ZkClient zk = new ZkClient(serverstring );
List<String> partitions = zk.getChildren("/brokers/topics/test-topic/1");
// my topic = test-topic, broker id = 1
This broker is associated with partition 0. So I am expecting output should
be [0] but I am getting empty response [].
Is there anything wrong in the way I am calling it?
Please help.
Thanks
Sourabh
On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Sourabh,
>
> In 0.7, you can use zkclient or the zookeeper client library to query the
> following zookeeper path to get a number of partitions for a topic -
>
> get /brokers/topics/[topic]/[broker-id]
>
> This will give you # of partitions/broker, you can then add up this value
> across all brokers.
>
> Thanks,
> Neha
>
>
> On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <chaki.sourabh@gmail.com
> >wrote:
>
> > Hi All,
> >
> > I am using zookeeper based connector to get data from Kafka Server. Is
> > there any way to get all the partitions from zookeeper for a given topic?
> >
> > I need do do that through java code.
> >
> > Please help!!!
> >
> > Thanks.
> >
>
Re: Get all the partion details from zookeeper for a topic
Posted by Neha Narkhede <ne...@gmail.com>.
You can instantiate ZkClient with the ZkStringSerializer in Kafka -
zkClient = new ZkClient(params.zkConnect, 30000, 30000,
ZKStringSerializer)
For code examples on how to read the various paths, you can take a look at
kafka.utils.ZkUtils.scala
Thanks,
Neha
On Wed, Mar 6, 2013 at 11:43 AM, sourabh chaki <ch...@gmail.com>wrote:
> Thanks Neha for your response.
> Now I am calling it like:
> List<String> partitions =
> zkclient.readData("/brokers/topics/test-topic/1");
> //topic = test topic and broker id is 1
>
> *1. * Now _connection.readData(path, stat, watch); is returning a
> byte[] of [49]. But derializable(data) is throwing ZkMarshallingError.
>
> org.I0Itec.zkclient.exception.ZkMarshallingError: java.io.EOFException
> at
>
> org.I0Itec.zkclient.serialize.SerializableSerializer.deserialize(SerializableSerializer.java:37)
>
> *2.* Apart from that byte 49 is number 1. But here my partition id is
> 0. Is derializable(data) suppose to convert [49] to [0] ?
>
>
> Code snippet where it is failing.
> Class ZkClient.java
>
> protected <T extends Object> T readData(final String path, final Stat
> stat, final boolean watch) {
> byte[] data = retryUntilConnected(new Callable<byte[]>() {
>
> @Override
> public byte[] call() throws Exception {
> return _connection.readData(path, stat, watch);
> }
> });
> return (T) derializable(data); //data here [49]. But derializable
> throwing ZkMarshallingError
> }
>
> Please help.
>
> Thanks
> Sourabh
>
>
> On Wed, Mar 6, 2013 at 10:20 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > I didn't check before I typed. I was referring to readData().
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Mar 5, 2013 at 11:04 PM, sourabh chaki <chaki.sourabh@gmail.com
> > >wrote:
> >
> > > Hi Neha,
> > > I am using zkclient version 0.1. I dont see any method "getData" in
> > > zkclient. Please let me know which version should I use.
> > >
> > > Thanks
> > > Sourabh
> > >
> > >
> > > On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > The problem is that you need to invoke getData, not getChildren on
> this
> > > > path - /brokers/topics/test-topic/1. That will return the # of
> > partitions
> > > > on this broker, which is 1.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <
> chaki.sourabh@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi Neha,
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > ZkClient zk = new ZkClient(serverstring );
> > > > > List<String> partitions =
> > > zk.getChildren("/brokers/topics/test-topic/1");
> > > > > // my topic = test-topic, broker id = 1
> > > > >
> > > > > This broker is associated with partition 0. So I am expecting
> output
> > > > should
> > > > > be [0] but I am getting empty response [].
> > > > >
> > > > > Is there anything wrong in the way I am calling it?
> > > > > Please help.
> > > > >
> > > > > Thanks
> > > > > Sourabh
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <
> > > neha.narkhede@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Sourabh,
> > > > > >
> > > > > > In 0.7, you can use zkclient or the zookeeper client library to
> > query
> > > > the
> > > > > > following zookeeper path to get a number of partitions for a
> topic
> > -
> > > > > >
> > > > > > get /brokers/topics/[topic]/[broker-id]
> > > > > >
> > > > > > This will give you # of partitions/broker, you can then add up
> this
> > > > value
> > > > > > across all brokers.
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> > > > chaki.sourabh@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I am using zookeeper based connector to get data from Kafka
> > Server.
> > > > Is
> > > > > > > there any way to get all the partitions from zookeeper for a
> > given
> > > > > topic?
> > > > > > >
> > > > > > > I need do do that through java code.
> > > > > > >
> > > > > > > Please help!!!
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Get all the partion details from zookeeper for a topic
Posted by sourabh chaki <ch...@gmail.com>.
Thanks Neha for your response.
Now I am calling it like:
List<String> partitions = zkclient.readData("/brokers/topics/test-topic/1");
//topic = test topic and broker id is 1
*1. * Now _connection.readData(path, stat, watch); is returning a
byte[] of [49]. But derializable(data) is throwing ZkMarshallingError.
org.I0Itec.zkclient.exception.ZkMarshallingError: java.io.EOFException
at
org.I0Itec.zkclient.serialize.SerializableSerializer.deserialize(SerializableSerializer.java:37)
*2.* Apart from that byte 49 is number 1. But here my partition id is
0. Is derializable(data) suppose to convert [49] to [0] ?
Code snippet where it is failing.
Class ZkClient.java
protected <T extends Object> T readData(final String path, final Stat
stat, final boolean watch) {
byte[] data = retryUntilConnected(new Callable<byte[]>() {
@Override
public byte[] call() throws Exception {
return _connection.readData(path, stat, watch);
}
});
return (T) derializable(data); //data here [49]. But derializable
throwing ZkMarshallingError
}
Please help.
Thanks
Sourabh
On Wed, Mar 6, 2013 at 10:20 PM, Neha Narkhede <ne...@gmail.com>wrote:
> I didn't check before I typed. I was referring to readData().
>
> Thanks,
> Neha
>
>
> On Tue, Mar 5, 2013 at 11:04 PM, sourabh chaki <chaki.sourabh@gmail.com
> >wrote:
>
> > Hi Neha,
> > I am using zkclient version 0.1. I dont see any method "getData" in
> > zkclient. Please let me know which version should I use.
> >
> > Thanks
> > Sourabh
> >
> >
> > On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > The problem is that you need to invoke getData, not getChildren on this
> > > path - /brokers/topics/test-topic/1. That will return the # of
> partitions
> > > on this broker, which is 1.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <chaki.sourabh@gmail.com
> > > >wrote:
> > >
> > > > Hi Neha,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > ZkClient zk = new ZkClient(serverstring );
> > > > List<String> partitions =
> > zk.getChildren("/brokers/topics/test-topic/1");
> > > > // my topic = test-topic, broker id = 1
> > > >
> > > > This broker is associated with partition 0. So I am expecting output
> > > should
> > > > be [0] but I am getting empty response [].
> > > >
> > > > Is there anything wrong in the way I am calling it?
> > > > Please help.
> > > >
> > > > Thanks
> > > > Sourabh
> > > >
> > > >
> > > > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <
> > neha.narkhede@gmail.com
> > > > >wrote:
> > > >
> > > > > Sourabh,
> > > > >
> > > > > In 0.7, you can use zkclient or the zookeeper client library to
> query
> > > the
> > > > > following zookeeper path to get a number of partitions for a topic
> -
> > > > >
> > > > > get /brokers/topics/[topic]/[broker-id]
> > > > >
> > > > > This will give you # of partitions/broker, you can then add up this
> > > value
> > > > > across all brokers.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> > > chaki.sourabh@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I am using zookeeper based connector to get data from Kafka
> Server.
> > > Is
> > > > > > there any way to get all the partitions from zookeeper for a
> given
> > > > topic?
> > > > > >
> > > > > > I need do do that through java code.
> > > > > >
> > > > > > Please help!!!
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Get all the partion details from zookeeper for a topic
Posted by Neha Narkhede <ne...@gmail.com>.
I didn't check before I typed. I was referring to readData().
Thanks,
Neha
On Tue, Mar 5, 2013 at 11:04 PM, sourabh chaki <ch...@gmail.com>wrote:
> Hi Neha,
> I am using zkclient version 0.1. I dont see any method "getData" in
> zkclient. Please let me know which version should I use.
>
> Thanks
> Sourabh
>
>
> On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > The problem is that you need to invoke getData, not getChildren on this
> > path - /brokers/topics/test-topic/1. That will return the # of partitions
> > on this broker, which is 1.
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <chaki.sourabh@gmail.com
> > >wrote:
> >
> > > Hi Neha,
> > >
> > > Thanks for your reply.
> > >
> > > ZkClient zk = new ZkClient(serverstring );
> > > List<String> partitions =
> zk.getChildren("/brokers/topics/test-topic/1");
> > > // my topic = test-topic, broker id = 1
> > >
> > > This broker is associated with partition 0. So I am expecting output
> > should
> > > be [0] but I am getting empty response [].
> > >
> > > Is there anything wrong in the way I am calling it?
> > > Please help.
> > >
> > > Thanks
> > > Sourabh
> > >
> > >
> > > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Sourabh,
> > > >
> > > > In 0.7, you can use zkclient or the zookeeper client library to query
> > the
> > > > following zookeeper path to get a number of partitions for a topic -
> > > >
> > > > get /brokers/topics/[topic]/[broker-id]
> > > >
> > > > This will give you # of partitions/broker, you can then add up this
> > value
> > > > across all brokers.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> > chaki.sourabh@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am using zookeeper based connector to get data from Kafka Server.
> > Is
> > > > > there any way to get all the partitions from zookeeper for a given
> > > topic?
> > > > >
> > > > > I need do do that through java code.
> > > > >
> > > > > Please help!!!
> > > > >
> > > > > Thanks.
> > > > >
> > > >
> > >
> >
>
Re: Get all the partion details from zookeeper for a topic
Posted by sourabh chaki <ch...@gmail.com>.
Hi Neha,
I am using zkclient version 0.1. I dont see any method "getData" in
zkclient. Please let me know which version should I use.
Thanks
Sourabh
On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <ne...@gmail.com>wrote:
> The problem is that you need to invoke getData, not getChildren on this
> path - /brokers/topics/test-topic/1. That will return the # of partitions
> on this broker, which is 1.
>
> Thanks,
> Neha
>
>
> On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <chaki.sourabh@gmail.com
> >wrote:
>
> > Hi Neha,
> >
> > Thanks for your reply.
> >
> > ZkClient zk = new ZkClient(serverstring );
> > List<String> partitions = zk.getChildren("/brokers/topics/test-topic/1");
> > // my topic = test-topic, broker id = 1
> >
> > This broker is associated with partition 0. So I am expecting output
> should
> > be [0] but I am getting empty response [].
> >
> > Is there anything wrong in the way I am calling it?
> > Please help.
> >
> > Thanks
> > Sourabh
> >
> >
> > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Sourabh,
> > >
> > > In 0.7, you can use zkclient or the zookeeper client library to query
> the
> > > following zookeeper path to get a number of partitions for a topic -
> > >
> > > get /brokers/topics/[topic]/[broker-id]
> > >
> > > This will give you # of partitions/broker, you can then add up this
> value
> > > across all brokers.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> chaki.sourabh@gmail.com
> > > >wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am using zookeeper based connector to get data from Kafka Server.
> Is
> > > > there any way to get all the partitions from zookeeper for a given
> > topic?
> > > >
> > > > I need do do that through java code.
> > > >
> > > > Please help!!!
> > > >
> > > > Thanks.
> > > >
> > >
> >
>
Re: Get all the partion details from zookeeper for a topic
Posted by Neha Narkhede <ne...@gmail.com>.
The problem is that you need to invoke getData, not getChildren on this
path - /brokers/topics/test-topic/1. That will return the # of partitions
on this broker, which is 1.
Thanks,
Neha
On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <ch...@gmail.com>wrote:
> Hi Neha,
>
> Thanks for your reply.
>
> ZkClient zk = new ZkClient(serverstring );
> List<String> partitions = zk.getChildren("/brokers/topics/test-topic/1");
> // my topic = test-topic, broker id = 1
>
> This broker is associated with partition 0. So I am expecting output should
> be [0] but I am getting empty response [].
>
> Is there anything wrong in the way I am calling it?
> Please help.
>
> Thanks
> Sourabh
>
>
> On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Sourabh,
> >
> > In 0.7, you can use zkclient or the zookeeper client library to query the
> > following zookeeper path to get a number of partitions for a topic -
> >
> > get /brokers/topics/[topic]/[broker-id]
> >
> > This will give you # of partitions/broker, you can then add up this value
> > across all brokers.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <chaki.sourabh@gmail.com
> > >wrote:
> >
> > > Hi All,
> > >
> > > I am using zookeeper based connector to get data from Kafka Server. Is
> > > there any way to get all the partitions from zookeeper for a given
> topic?
> > >
> > > I need do do that through java code.
> > >
> > > Please help!!!
> > >
> > > Thanks.
> > >
> >
>