You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Mark Nuttall <mk...@gmail.com> on 2021/04/16 15:09:34 UTC
kafka consumer to kafka producer parition key issue
Camel version: 3.9.0
Issue: when consuming from one kafka topic and producing to another,
setting the partition key causes the write to fail. After a few seconds it
will fail with this error:
Error during processing. Exchange[2729AA824AE4E97-0000000000000000]. Caused
by: [org.apache.kafka.common.errors.TimeoutException - Topic some-topic not
present in metadata after 60000 ms.]
The topic does exist. If i produce to the topic without the partition key
OR I produce to the topic with a partition key via the producerTemplate it
works just fine.
I am pretty sure in the past i have done this and it worked.
Has anyone else seen this? Any ideas? I do have a work around (i.e. use
Spring's kafkaTemplate.).
Mark
Re: kafka consumer to kafka producer parition key issue
Posted by Mark Nuttall <mk...@gmail.com>.
Thanks Omar. I was looking at ProducerRecord because I was thinking about
trying to use Spring Kafka.
I am not sure yet, but part of the issue is that in the camel code, what is
being called the paritionKey is not. It is the partition (number). The docs
do the same thing. The partitionKey is a value that is converted by kafka
to determine the partition.
Ok so, yeah. I tested it. Setting just the KafkaConstants.KEY works.
There is no Kafka partition key and message key. There is only "Key". It
is used for partitioning and compaction.
Only use the value in Camel called "KafkaConstants.PARTITION_KEY" if you
want to set it to a particular partition by number. The docs are correct in
that it is the partition the message will be assigned to. There is also
constant called KafkaConstants.PARTITION but it is for consumers.
Thanks for helping me think through this Omar.
Mark
On Mon, Apr 19, 2021 at 12:05 PM Omar Al-Safi <om...@oalsafi.com> wrote:
> Weird, from experience, I have seen this timeout happen if the specified
> partition doesn't exist.
>
> However, since you are using Camel 3.9 and you didn't set a message key,
> the partition key won't be set in the ProducerRecord, it will be only set
> if you set the message key, however this behavior has changed in Camel 3.10
> where the partition key won't require setting the message key. Thus, even
> if you set the partition key but you didn't set a message key, I will
> expect camel-kafka to ignore the partition (based on Camel 3.9) as you can
> see in this code
> <
> https://github.com/apache/camel/blob/camel-3.9.0/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java#L275
> >
> .
>
> How about if you try using camel-vertx-kafka
> <https://camel.apache.org/components/3.9.x/vertx-kafka-component.html>
> component
> and see if this solves the problem?
>
> Regards,
> Omar
>
> On Mon, Apr 19, 2021 at 5:26 PM Mark Nuttall <mk...@gmail.com> wrote:
>
> > Setting the message key does not help. :(
> >
> > On Mon, Apr 19, 2021 at 11:20 AM Mark Nuttall <mk...@gmail.com>
> wrote:
> >
> > > I've never set the message key. I might give that a shot.
> > >
> > > The route is pretty simple. And the only headers being set are the
> > > partition key and another string value i need to be passed. The body is
> > > JSON.
> > > I am running Kafka locally (so whatever partitions is the default) and
> > AWS
> > > MSK - 2 partitions.
> > >
> > > Mark
> > >
> > >
> > >
> > > On Mon, Apr 19, 2021 at 4:14 AM Omar Al-Safi <om...@oalsafi.com> wrote:
> > >
> > >> Hi Mark,
> > >>
> > >> When you send your message to Kafka, do you as well set the message
> key
> > or
> > >> you just set the partition key? Perhaps it would be great if you can
> > post
> > >> an example for your route with the headers/body that you set, that
> could
> > >> help us here to troubleshoot the issue. Also, how many partitions do
> you
> > >> have in your topic?
> > >>
> > >> Regards,
> > >> Omar
> > >>
> > >> On Fri, Apr 16, 2021 at 5:10 PM Mark Nuttall <mk...@gmail.com>
> > wrote:
> > >>
> > >> > Camel version: 3.9.0
> > >> >
> > >> > Issue: when consuming from one kafka topic and producing to another,
> > >> > setting the partition key causes the write to fail. After a few
> > seconds
> > >> it
> > >> > will fail with this error:
> > >> >
> > >> > Error during processing. Exchange[2729AA824AE4E97-0000000000000000].
> > >> Caused
> > >> > by: [org.apache.kafka.common.errors.TimeoutException - Topic
> > some-topic
> > >> not
> > >> > present in metadata after 60000 ms.]
> > >> >
> > >> > The topic does exist. If i produce to the topic without the
> partition
> > >> key
> > >> > OR I produce to the topic with a partition key via the
> > producerTemplate
> > >> it
> > >> > works just fine.
> > >> > I am pretty sure in the past i have done this and it worked.
> > >> >
> > >> > Has anyone else seen this? Any ideas? I do have a work around (i.e.
> > use
> > >> > Spring's kafkaTemplate.).
> > >> >
> > >> > Mark
> > >> >
> > >>
> > >
> >
>
Re: kafka consumer to kafka producer parition key issue
Posted by Omar Al-Safi <om...@oalsafi.com>.
Weird, from experience, I have seen this timeout happen if the specified
partition doesn't exist.
However, since you are using Camel 3.9 and you didn't set a message key,
the partition key won't be set in the ProducerRecord, it will be only set
if you set the message key, however this behavior has changed in Camel 3.10
where the partition key won't require setting the message key. Thus, even
if you set the partition key but you didn't set a message key, I will
expect camel-kafka to ignore the partition (based on Camel 3.9) as you can
see in this code
<https://github.com/apache/camel/blob/camel-3.9.0/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java#L275>
.
How about if you try using camel-vertx-kafka
<https://camel.apache.org/components/3.9.x/vertx-kafka-component.html>
component
and see if this solves the problem?
Regards,
Omar
On Mon, Apr 19, 2021 at 5:26 PM Mark Nuttall <mk...@gmail.com> wrote:
> Setting the message key does not help. :(
>
> On Mon, Apr 19, 2021 at 11:20 AM Mark Nuttall <mk...@gmail.com> wrote:
>
> > I've never set the message key. I might give that a shot.
> >
> > The route is pretty simple. And the only headers being set are the
> > partition key and another string value i need to be passed. The body is
> > JSON.
> > I am running Kafka locally (so whatever partitions is the default) and
> AWS
> > MSK - 2 partitions.
> >
> > Mark
> >
> >
> >
> > On Mon, Apr 19, 2021 at 4:14 AM Omar Al-Safi <om...@oalsafi.com> wrote:
> >
> >> Hi Mark,
> >>
> >> When you send your message to Kafka, do you as well set the message key
> or
> >> you just set the partition key? Perhaps it would be great if you can
> post
> >> an example for your route with the headers/body that you set, that could
> >> help us here to troubleshoot the issue. Also, how many partitions do you
> >> have in your topic?
> >>
> >> Regards,
> >> Omar
> >>
> >> On Fri, Apr 16, 2021 at 5:10 PM Mark Nuttall <mk...@gmail.com>
> wrote:
> >>
> >> > Camel version: 3.9.0
> >> >
> >> > Issue: when consuming from one kafka topic and producing to another,
> >> > setting the partition key causes the write to fail. After a few
> seconds
> >> it
> >> > will fail with this error:
> >> >
> >> > Error during processing. Exchange[2729AA824AE4E97-0000000000000000].
> >> Caused
> >> > by: [org.apache.kafka.common.errors.TimeoutException - Topic
> some-topic
> >> not
> >> > present in metadata after 60000 ms.]
> >> >
> >> > The topic does exist. If i produce to the topic without the partition
> >> key
> >> > OR I produce to the topic with a partition key via the
> producerTemplate
> >> it
> >> > works just fine.
> >> > I am pretty sure in the past i have done this and it worked.
> >> >
> >> > Has anyone else seen this? Any ideas? I do have a work around (i.e.
> use
> >> > Spring's kafkaTemplate.).
> >> >
> >> > Mark
> >> >
> >>
> >
>
Re: kafka consumer to kafka producer parition key issue
Posted by Mark Nuttall <mk...@gmail.com>.
Setting the message key does not help. :(
On Mon, Apr 19, 2021 at 11:20 AM Mark Nuttall <mk...@gmail.com> wrote:
> I've never set the message key. I might give that a shot.
>
> The route is pretty simple. And the only headers being set are the
> partition key and another string value i need to be passed. The body is
> JSON.
> I am running Kafka locally (so whatever partitions is the default) and AWS
> MSK - 2 partitions.
>
> Mark
>
>
>
> On Mon, Apr 19, 2021 at 4:14 AM Omar Al-Safi <om...@oalsafi.com> wrote:
>
>> Hi Mark,
>>
>> When you send your message to Kafka, do you as well set the message key or
>> you just set the partition key? Perhaps it would be great if you can post
>> an example for your route with the headers/body that you set, that could
>> help us here to troubleshoot the issue. Also, how many partitions do you
>> have in your topic?
>>
>> Regards,
>> Omar
>>
>> On Fri, Apr 16, 2021 at 5:10 PM Mark Nuttall <mk...@gmail.com> wrote:
>>
>> > Camel version: 3.9.0
>> >
>> > Issue: when consuming from one kafka topic and producing to another,
>> > setting the partition key causes the write to fail. After a few seconds
>> it
>> > will fail with this error:
>> >
>> > Error during processing. Exchange[2729AA824AE4E97-0000000000000000].
>> Caused
>> > by: [org.apache.kafka.common.errors.TimeoutException - Topic some-topic
>> not
>> > present in metadata after 60000 ms.]
>> >
>> > The topic does exist. If i produce to the topic without the partition
>> key
>> > OR I produce to the topic with a partition key via the producerTemplate
>> it
>> > works just fine.
>> > I am pretty sure in the past i have done this and it worked.
>> >
>> > Has anyone else seen this? Any ideas? I do have a work around (i.e. use
>> > Spring's kafkaTemplate.).
>> >
>> > Mark
>> >
>>
>
Re: kafka consumer to kafka producer parition key issue
Posted by Mark Nuttall <mk...@gmail.com>.
I've never set the message key. I might give that a shot.
The route is pretty simple. And the only headers being set are the
partition key and another string value i need to be passed. The body is
JSON.
I am running Kafka locally (so whatever partitions is the default) and AWS
MSK - 2 partitions.
Mark
On Mon, Apr 19, 2021 at 4:14 AM Omar Al-Safi <om...@oalsafi.com> wrote:
> Hi Mark,
>
> When you send your message to Kafka, do you as well set the message key or
> you just set the partition key? Perhaps it would be great if you can post
> an example for your route with the headers/body that you set, that could
> help us here to troubleshoot the issue. Also, how many partitions do you
> have in your topic?
>
> Regards,
> Omar
>
> On Fri, Apr 16, 2021 at 5:10 PM Mark Nuttall <mk...@gmail.com> wrote:
>
> > Camel version: 3.9.0
> >
> > Issue: when consuming from one kafka topic and producing to another,
> > setting the partition key causes the write to fail. After a few seconds
> it
> > will fail with this error:
> >
> > Error during processing. Exchange[2729AA824AE4E97-0000000000000000].
> Caused
> > by: [org.apache.kafka.common.errors.TimeoutException - Topic some-topic
> not
> > present in metadata after 60000 ms.]
> >
> > The topic does exist. If i produce to the topic without the partition key
> > OR I produce to the topic with a partition key via the producerTemplate
> it
> > works just fine.
> > I am pretty sure in the past i have done this and it worked.
> >
> > Has anyone else seen this? Any ideas? I do have a work around (i.e. use
> > Spring's kafkaTemplate.).
> >
> > Mark
> >
>
Re: kafka consumer to kafka producer parition key issue
Posted by Omar Al-Safi <om...@oalsafi.com>.
Hi Mark,
When you send your message to Kafka, do you as well set the message key or
you just set the partition key? Perhaps it would be great if you can post
an example for your route with the headers/body that you set, that could
help us here to troubleshoot the issue. Also, how many partitions do you
have in your topic?
Regards,
Omar
On Fri, Apr 16, 2021 at 5:10 PM Mark Nuttall <mk...@gmail.com> wrote:
> Camel version: 3.9.0
>
> Issue: when consuming from one kafka topic and producing to another,
> setting the partition key causes the write to fail. After a few seconds it
> will fail with this error:
>
> Error during processing. Exchange[2729AA824AE4E97-0000000000000000]. Caused
> by: [org.apache.kafka.common.errors.TimeoutException - Topic some-topic not
> present in metadata after 60000 ms.]
>
> The topic does exist. If i produce to the topic without the partition key
> OR I produce to the topic with a partition key via the producerTemplate it
> works just fine.
> I am pretty sure in the past i have done this and it worked.
>
> Has anyone else seen this? Any ideas? I do have a work around (i.e. use
> Spring's kafkaTemplate.).
>
> Mark
>