You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by David Morales de Frías <dm...@paradigmatecnologico.com> on 2014/02/25 12:01:54 UTC
calling ad-hoc topic creation from code
Hi there,
I'm trying to create a topic from java code, by calling CreateTopicCommand:
*ZkClient zkClient = new ZkClient(zookeeperCluster, 30000, 30000);*
*CreateTopicCommand.createTopic(zkClient, topic,
numPartitions.intValue(),replicationFactor.intValue(), "");*
*zkClient.close();*
The program runs without errors and even if i call it twice, the second
time i receive an exception telling me that this topic already exists.
But, if run a list-topics, the topic doesn't appear... and there is no
folder under kafka-logs
If i want to create it by using the create-topic script, it prints that the
topic already exists¡
Any idea about that?
Regards.
Re: calling ad-hoc topic creation from code
Posted by David Morales de Frías <dm...@paradigmatecnologico.com>.
[solved]. zkClient needs a serializer. This is the final code:
ZkClient zkClient = new ZkClient(zookeeperCluster, 30000, 30000);
zkClient.setZkSerializer(new ZkSerializer() {
@Override
public byte[] serialize(Object o)
throws ZkMarshallingError
{
return ZKStringSerializer.serialize(o);
}
@Override
public Object deserialize(byte[] bytes)
throws ZkMarshallingError
{
return ZKStringSerializer.deserialize(bytes);
}
});
CreateTopicCommand.createTopic(zkClient, topic, numPartitions.intValue(),
replicationFactor.intValue(), "");
zkClient.close();
Regards.
2014-02-25 18:50 GMT+01:00 David Morales de Frías <
dmorales@paradigmatecnologico.com>:
> Nope... i have checked this and replication factor is 1. Anyway, when i
> tried to increase this param, an exception is thrown (rep. factor greater
> than num. of brokers).
>
> There is no difference in this code and the script, apparently, i can only
> see changes in the constructor used in zkClient.
>
> Any clue?
>
>
> Thanks.
>
>
> 2014-02-25 18:41 GMT+01:00 Neha Narkhede <ne...@gmail.com>:
>
> David,
>>
>> Topic creation can fail if you specify the replication factor > # of
>> brokers in the cluster. Can you check if that is true in your case?
>> Unfortunately, I don't think we fail the createTopic() API with the
>> appropriate exception since there is still a race condition where the
>> broker can come up after you do the check and before you throw the
>> exception. We are trying to fix this behavior in one of the upcoming
>> releases to let topic creation succeed even if the # of live brokers in
>> the
>> cluster is less than the replication factor at topic creation time.
>>
>> Thanks,
>> Neha
>>
>>
>> On Tue, Feb 25, 2014 at 8:39 AM, David Morales de Frías <
>> dmorales@paradigmatecnologico.com> wrote:
>>
>> > This is the code that i can see in CreateTopicCommand
>> >
>> >
>> > var zkClient: ZkClient = null
>> > try {
>> > zkClient = new ZkClient(zkConnect, 30000, 30000,
>> ZKStringSerializer)
>> > createTopic(zkClient, topic, nPartitions, replicationFactor,
>> > replicaAssignmentStr)
>> > println("creation succeeded!")
>> > } catch {
>> > case e: Throwable =>
>> > println("creation failed because of " + e.getMessage)
>> > println(Utils.stackTrace(e))
>> > } finally {
>> > if (zkClient != null)
>> > zkClient.close()
>> > }
>> >
>> >
>> > Mine is *ZkClient zkClient = new ZkClient(zookeeperCluster, 30000,
>> 30000);*
>> >
>> >
>> >
>> > Thanks
>> >
>> >
>> >
>> > 2014-02-25 17:25 GMT+01:00 Jun Rao <ju...@gmail.com>:
>> >
>> > > Is the ZK connection string + namespace the same btw the code and the
>> > > script?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Tue, Feb 25, 2014 at 3:01 AM, David Morales de Frías <
>> > > dmorales@paradigmatecnologico.com> wrote:
>> > >
>> > > > Hi there,
>> > > >
>> > > > I'm trying to create a topic from java code, by calling
>> > > CreateTopicCommand:
>> > > >
>> > > >
>> > > > *ZkClient zkClient = new ZkClient(zookeeperCluster, 30000, 30000);*
>> > > >
>> > > > *CreateTopicCommand.createTopic(zkClient, topic,
>> > > > numPartitions.intValue(),replicationFactor.intValue(), "");*
>> > > >
>> > > > *zkClient.close();*
>> > > >
>> > > >
>> > > > The program runs without errors and even if i call it twice, the
>> second
>> > > > time i receive an exception telling me that this topic already
>> exists.
>> > > >
>> > > > But, if run a list-topics, the topic doesn't appear... and there is
>> no
>> > > > folder under kafka-logs
>> > > >
>> > > > If i want to create it by using the create-topic script, it prints
>> that
>> > > the
>> > > > topic already exists¡
>> > > >
>> > > > Any idea about that?
>> > > >
>> > > >
>> > > >
>> > > > Regards.
>> > > >
>> > >
>> >
>>
>
>
Re: calling ad-hoc topic creation from code
Posted by David Morales de Frías <dm...@paradigmatecnologico.com>.
Nope... i have checked this and replication factor is 1. Anyway, when i
tried to increase this param, an exception is thrown (rep. factor greater
than num. of brokers).
There is no difference in this code and the script, apparently, i can only
see changes in the constructor used in zkClient.
Any clue?
Thanks.
2014-02-25 18:41 GMT+01:00 Neha Narkhede <ne...@gmail.com>:
> David,
>
> Topic creation can fail if you specify the replication factor > # of
> brokers in the cluster. Can you check if that is true in your case?
> Unfortunately, I don't think we fail the createTopic() API with the
> appropriate exception since there is still a race condition where the
> broker can come up after you do the check and before you throw the
> exception. We are trying to fix this behavior in one of the upcoming
> releases to let topic creation succeed even if the # of live brokers in the
> cluster is less than the replication factor at topic creation time.
>
> Thanks,
> Neha
>
>
> On Tue, Feb 25, 2014 at 8:39 AM, David Morales de Frías <
> dmorales@paradigmatecnologico.com> wrote:
>
> > This is the code that i can see in CreateTopicCommand
> >
> >
> > var zkClient: ZkClient = null
> > try {
> > zkClient = new ZkClient(zkConnect, 30000, 30000,
> ZKStringSerializer)
> > createTopic(zkClient, topic, nPartitions, replicationFactor,
> > replicaAssignmentStr)
> > println("creation succeeded!")
> > } catch {
> > case e: Throwable =>
> > println("creation failed because of " + e.getMessage)
> > println(Utils.stackTrace(e))
> > } finally {
> > if (zkClient != null)
> > zkClient.close()
> > }
> >
> >
> > Mine is *ZkClient zkClient = new ZkClient(zookeeperCluster, 30000,
> 30000);*
> >
> >
> >
> > Thanks
> >
> >
> >
> > 2014-02-25 17:25 GMT+01:00 Jun Rao <ju...@gmail.com>:
> >
> > > Is the ZK connection string + namespace the same btw the code and the
> > > script?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Feb 25, 2014 at 3:01 AM, David Morales de Frías <
> > > dmorales@paradigmatecnologico.com> wrote:
> > >
> > > > Hi there,
> > > >
> > > > I'm trying to create a topic from java code, by calling
> > > CreateTopicCommand:
> > > >
> > > >
> > > > *ZkClient zkClient = new ZkClient(zookeeperCluster, 30000, 30000);*
> > > >
> > > > *CreateTopicCommand.createTopic(zkClient, topic,
> > > > numPartitions.intValue(),replicationFactor.intValue(), "");*
> > > >
> > > > *zkClient.close();*
> > > >
> > > >
> > > > The program runs without errors and even if i call it twice, the
> second
> > > > time i receive an exception telling me that this topic already
> exists.
> > > >
> > > > But, if run a list-topics, the topic doesn't appear... and there is
> no
> > > > folder under kafka-logs
> > > >
> > > > If i want to create it by using the create-topic script, it prints
> that
> > > the
> > > > topic already exists¡
> > > >
> > > > Any idea about that?
> > > >
> > > >
> > > >
> > > > Regards.
> > > >
> > >
> >
>
Re: calling ad-hoc topic creation from code
Posted by Neha Narkhede <ne...@gmail.com>.
David,
Topic creation can fail if you specify the replication factor > # of
brokers in the cluster. Can you check if that is true in your case?
Unfortunately, I don't think we fail the createTopic() API with the
appropriate exception since there is still a race condition where the
broker can come up after you do the check and before you throw the
exception. We are trying to fix this behavior in one of the upcoming
releases to let topic creation succeed even if the # of live brokers in the
cluster is less than the replication factor at topic creation time.
Thanks,
Neha
On Tue, Feb 25, 2014 at 8:39 AM, David Morales de Frías <
dmorales@paradigmatecnologico.com> wrote:
> This is the code that i can see in CreateTopicCommand
>
>
> var zkClient: ZkClient = null
> try {
> zkClient = new ZkClient(zkConnect, 30000, 30000, ZKStringSerializer)
> createTopic(zkClient, topic, nPartitions, replicationFactor,
> replicaAssignmentStr)
> println("creation succeeded!")
> } catch {
> case e: Throwable =>
> println("creation failed because of " + e.getMessage)
> println(Utils.stackTrace(e))
> } finally {
> if (zkClient != null)
> zkClient.close()
> }
>
>
> Mine is *ZkClient zkClient = new ZkClient(zookeeperCluster, 30000, 30000);*
>
>
>
> Thanks
>
>
>
> 2014-02-25 17:25 GMT+01:00 Jun Rao <ju...@gmail.com>:
>
> > Is the ZK connection string + namespace the same btw the code and the
> > script?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 25, 2014 at 3:01 AM, David Morales de Frías <
> > dmorales@paradigmatecnologico.com> wrote:
> >
> > > Hi there,
> > >
> > > I'm trying to create a topic from java code, by calling
> > CreateTopicCommand:
> > >
> > >
> > > *ZkClient zkClient = new ZkClient(zookeeperCluster, 30000, 30000);*
> > >
> > > *CreateTopicCommand.createTopic(zkClient, topic,
> > > numPartitions.intValue(),replicationFactor.intValue(), "");*
> > >
> > > *zkClient.close();*
> > >
> > >
> > > The program runs without errors and even if i call it twice, the second
> > > time i receive an exception telling me that this topic already exists.
> > >
> > > But, if run a list-topics, the topic doesn't appear... and there is no
> > > folder under kafka-logs
> > >
> > > If i want to create it by using the create-topic script, it prints that
> > the
> > > topic already exists¡
> > >
> > > Any idea about that?
> > >
> > >
> > >
> > > Regards.
> > >
> >
>
Re: calling ad-hoc topic creation from code
Posted by David Morales de Frías <dm...@paradigmatecnologico.com>.
This is the code that i can see in CreateTopicCommand
var zkClient: ZkClient = null
try {
zkClient = new ZkClient(zkConnect, 30000, 30000, ZKStringSerializer)
createTopic(zkClient, topic, nPartitions, replicationFactor,
replicaAssignmentStr)
println("creation succeeded!")
} catch {
case e: Throwable =>
println("creation failed because of " + e.getMessage)
println(Utils.stackTrace(e))
} finally {
if (zkClient != null)
zkClient.close()
}
Mine is *ZkClient zkClient = new ZkClient(zookeeperCluster, 30000, 30000);*
Thanks
2014-02-25 17:25 GMT+01:00 Jun Rao <ju...@gmail.com>:
> Is the ZK connection string + namespace the same btw the code and the
> script?
>
> Thanks,
>
> Jun
>
>
> On Tue, Feb 25, 2014 at 3:01 AM, David Morales de Frías <
> dmorales@paradigmatecnologico.com> wrote:
>
> > Hi there,
> >
> > I'm trying to create a topic from java code, by calling
> CreateTopicCommand:
> >
> >
> > *ZkClient zkClient = new ZkClient(zookeeperCluster, 30000, 30000);*
> >
> > *CreateTopicCommand.createTopic(zkClient, topic,
> > numPartitions.intValue(),replicationFactor.intValue(), "");*
> >
> > *zkClient.close();*
> >
> >
> > The program runs without errors and even if i call it twice, the second
> > time i receive an exception telling me that this topic already exists.
> >
> > But, if run a list-topics, the topic doesn't appear... and there is no
> > folder under kafka-logs
> >
> > If i want to create it by using the create-topic script, it prints that
> the
> > topic already exists¡
> >
> > Any idea about that?
> >
> >
> >
> > Regards.
> >
>
Re: calling ad-hoc topic creation from code
Posted by Jun Rao <ju...@gmail.com>.
Is the ZK connection string + namespace the same btw the code and the
script?
Thanks,
Jun
On Tue, Feb 25, 2014 at 3:01 AM, David Morales de Frías <
dmorales@paradigmatecnologico.com> wrote:
> Hi there,
>
> I'm trying to create a topic from java code, by calling CreateTopicCommand:
>
>
> *ZkClient zkClient = new ZkClient(zookeeperCluster, 30000, 30000);*
>
> *CreateTopicCommand.createTopic(zkClient, topic,
> numPartitions.intValue(),replicationFactor.intValue(), "");*
>
> *zkClient.close();*
>
>
> The program runs without errors and even if i call it twice, the second
> time i receive an exception telling me that this topic already exists.
>
> But, if run a list-topics, the topic doesn't appear... and there is no
> folder under kafka-logs
>
> If i want to create it by using the create-topic script, it prints that the
> topic already exists¡
>
> Any idea about that?
>
>
>
> Regards.
>