Kafka Streams for Remote Server

[mishadoff <mi...@gmail.com>]

Hello,

I’ve a simplest ever kafka streams application which just reads from one kafka topic A and write to another topic B.

When I run it on my local environment (local zk, local kafka broker, local kafka streams app) everything works fine, topic B created and filled with messages from A
If I run it on existing kafka cluster (remote zk, remote kafka, LOCAL kafka streams) my app is not working anymore.

It succesfully read the remote topic A, succesfully process the message and generate a producer record, creates a B topic in remote kafka, bud during send I get an error

```
15:36:47.242 [kafka-producer-network-thread | example-message-counter3-1-StreamThread-1-producer] ERROR o.a.k.s.p.internals.RecordCollector - Error sending record: null
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition
```

Could you point me to direction where to start debug or what problems might cause this behaviour?

Thanks,
— Misha

Re: Kafka Streams for Remote Server

[Guozhang Wang <wa...@gmail.com>]

Hi Misha,

If it works locally, then I would still suspect that it is due to a
transient timing issue: note that the create-topic script is non-blocking,
i.e. even when it returns it does not necessarily guarantee that the leader
metadata information has been completed propagating to brokers.



Guozhang


On Thu, Jul 28, 2016 at 1:48 AM, mishadoff <mi...@gmail.com> wrote:

> Thanks Guozhang,
>
> Yes, I rely on auto-create, and it works locally. Maybe I need to tweak
> some timeout conf for that?
> Also I identified, that even I manually create a topic, it lists but I can
> not produce messages to this topic with the same exception.
> Producing to other topics works well, so it seems like server problem?
>
> — Misha
>
> > On Jul 28, 2016, at 01:44, Guozhang Wang <wa...@gmail.com> wrote:
> >
> > Misha,
> >
> > Did you pre-create the sink topic before starting your application or you
> > are relying on the broker-side auto-create for that topic?
> >
> > If you are relying on auto-create, then there is a transient period where
> > the topic is created but the metadata has not been propagated to the
> > brokers so they do not know they are the leader of the created topic
> > partitions yet. And I'd recommend not relying on it since it is really
> > meant for debugging environment only.
> >
> > Guozhang
> >
> >
> > On Wed, Jul 27, 2016 at 5:45 AM, mishadoff <mi...@gmail.com> wrote:
> >
> >> Hello,
> >>
> >> I’ve a simplest ever kafka streams application which just reads from one
> >> kafka topic A and write to another topic B.
> >>
> >> When I run it on my local environment (local zk, local kafka broker,
> local
> >> kafka streams app) everything works fine, topic B created and filled
> with
> >> messages from A
> >> If I run it on existing kafka cluster (remote zk, remote kafka, LOCAL
> >> kafka streams) my app is not working anymore.
> >>
> >> It succesfully read the remote topic A, succesfully process the message
> >> and generate a producer record, creates a B topic in remote kafka, bud
> >> during send I get an error
> >>
> >> ```
> >> 15:36:47.242 [kafka-producer-network-thread |
> >> example-message-counter3-1-StreamThread-1-producer] ERROR
> >> o.a.k.s.p.internals.RecordCollector - Error sending record: null
> >> org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> server
> >> is not the leader for that topic-partition
> >> ```
> >>
> >> Could you point me to direction where to start debug or what problems
> >> might cause this behaviour?
> >>
> >> Thanks,
> >> — Misha
> >
> >
> >
> >
> > --
> > -- Guozhang
>
>


-- 
-- Guozhang

Re: Kafka Streams for Remote Server

[mishadoff <mi...@gmail.com>]

Thanks Guozhang, 

Yes, I rely on auto-create, and it works locally. Maybe I need to tweak some timeout conf for that?
Also I identified, that even I manually create a topic, it lists but I can not produce messages to this topic with the same exception.
Producing to other topics works well, so it seems like server problem?

— Misha

> On Jul 28, 2016, at 01:44, Guozhang Wang <wa...@gmail.com> wrote:
> 
> Misha,
> 
> Did you pre-create the sink topic before starting your application or you
> are relying on the broker-side auto-create for that topic?
> 
> If you are relying on auto-create, then there is a transient period where
> the topic is created but the metadata has not been propagated to the
> brokers so they do not know they are the leader of the created topic
> partitions yet. And I'd recommend not relying on it since it is really
> meant for debugging environment only.
> 
> Guozhang
> 
> 
> On Wed, Jul 27, 2016 at 5:45 AM, mishadoff <mi...@gmail.com> wrote:
> 
>> Hello,
>> 
>> I’ve a simplest ever kafka streams application which just reads from one
>> kafka topic A and write to another topic B.
>> 
>> When I run it on my local environment (local zk, local kafka broker, local
>> kafka streams app) everything works fine, topic B created and filled with
>> messages from A
>> If I run it on existing kafka cluster (remote zk, remote kafka, LOCAL
>> kafka streams) my app is not working anymore.
>> 
>> It succesfully read the remote topic A, succesfully process the message
>> and generate a producer record, creates a B topic in remote kafka, bud
>> during send I get an error
>> 
>> ```
>> 15:36:47.242 [kafka-producer-network-thread |
>> example-message-counter3-1-StreamThread-1-producer] ERROR
>> o.a.k.s.p.internals.RecordCollector - Error sending record: null
>> org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
>> is not the leader for that topic-partition
>> ```
>> 
>> Could you point me to direction where to start debug or what problems
>> might cause this behaviour?
>> 
>> Thanks,
>> — Misha
> 
> 
> 
> 
> -- 
> -- Guozhang

Re: Kafka Streams for Remote Server

[Guozhang Wang <wa...@gmail.com>]

Misha,

Did you pre-create the sink topic before starting your application or you
are relying on the broker-side auto-create for that topic?

If you are relying on auto-create, then there is a transient period where
the topic is created but the metadata has not been propagated to the
brokers so they do not know they are the leader of the created topic
partitions yet. And I'd recommend not relying on it since it is really
meant for debugging environment only.

Guozhang


On Wed, Jul 27, 2016 at 5:45 AM, mishadoff <mi...@gmail.com> wrote:

> Hello,
>
> I’ve a simplest ever kafka streams application which just reads from one
> kafka topic A and write to another topic B.
>
> When I run it on my local environment (local zk, local kafka broker, local
> kafka streams app) everything works fine, topic B created and filled with
> messages from A
> If I run it on existing kafka cluster (remote zk, remote kafka, LOCAL
> kafka streams) my app is not working anymore.
>
> It succesfully read the remote topic A, succesfully process the message
> and generate a producer record, creates a B topic in remote kafka, bud
> during send I get an error
>
> ```
> 15:36:47.242 [kafka-producer-network-thread |
> example-message-counter3-1-StreamThread-1-producer] ERROR
> o.a.k.s.p.internals.RecordCollector - Error sending record: null
> org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
> is not the leader for that topic-partition
> ```
>
> Could you point me to direction where to start debug or what problems
> might cause this behaviour?
>
> Thanks,
> — Misha




-- 
-- Guozhang