You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by mark <mp...@gmail.com> on 2021/01/14 16:27:54 UTC
Camel-K / Kafka / Avro: something broke overnight
Hello,
Until yesterday we were using Camel-K to send Avro-encoded messages to
Kafka using code of the form,
.to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
+
"&keySerializerClass=org.apache.kafka.common.serialization.StringSerializer"
+
"&serializerClass=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
+ "&additionalProperties.apicurio.registry.url=" +
schemaRegistry
Running this today, our Integration had picked up camel-kafka 3.7.0 and
issue https://issues.apache.org/jira/browse/CAMEL-15770 which changed the
first two properties to keySerializer and valueSerializer. However
adjusting the code to,
.to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
+
"&keySerializer=org.apache.kafka.common.serialization.StringSerializer"
+
"&valueSerializer=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
+ "&additionalProperties.apicurio.registry.url=" + schemaRegistry
Leaves us no longer able to set additionalProperties. Attempts such as
adding
-d mvn:org.apache.camel/camel-kafka:[3.6.0]
have no effect: we always get camel-kafka 3.7 and cannot pin the
Integration to 3.6 or below. The relevant stack trace is below - please can
anyone help us either adjust the apicurio.registry.url for the new library,
or pin things back to 3.6 please?
[1] Caused by: org.apache.kafka.common.KafkaException: Failed to construct
kafka producer
[1] at
org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:434)
[1] at
org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)
[1] at
org.apache.camel.component.kafka.KafkaProducer.doStart(KafkaProducer.java:115)
[1] at
org.apache.camel.support.service.BaseService.start(BaseService.java:115)
[1] at
org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
[1] at
org.apache.camel.impl.engine.AbstractCamelContext.internalAddService(AbstractCamelContext.java:1425)
[1] at
org.apache.camel.impl.engine.AbstractCamelContext.addService(AbstractCamelContext.java:1343)
[1] at
org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:236)
[1] at
org.apache.camel.support.service.BaseService.start(BaseService.java:115)
[1] at
org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
[1] at
org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:101)
[1] at
org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1487)
[1] at
org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
[1] ... 37 more
[1] Caused by: java.lang.IllegalArgumentException: Missing registry base
url, set apicurio.registry.url
[1] at
io.apicurio.registry.utils.serde.AbstractKafkaSerDe.configure(AbstractKafkaSerDe.java:120)
[1] at
io.apicurio.registry.utils.serde.AbstractKafkaStrategyAwareSerDe.configure(AbstractKafkaStrategyAwareSerDe.java:75)
[1] at
io.apicurio.registry.utils.serde.AvroKafkaSerializer.configure(AvroKafkaSerializer.java:78)
[1] at
org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:369)
[1] ... 49 more
My dependencies are currently,
// camel-k:
dependency=mvn:org.apache.camel.quarkus/camel-quarkus-kafka:1.5.0
// camel-k: dependency=mvn:org.apache.avro/avro:1.10.1
// camel-k: dependency=mvn:org.glassfish.jersey.core/jersey-common:2.22.2
// camel-k:
dependency=mvn:io.apicurio/apicurio-registry-utils-serde:1.3.2.Final
Many thanks in advance,
Regards,
Mark
Re: Camel-K / Kafka / Avro: something broke overnight
Posted by mark <mp...@gmail.com>.
Andrea,
Thank you very much for the pointer. Yes we've hacked something
together using the test code you linked above, and a hand-rolled
deserializer based on
https://github.com/Apicurio/apicurio-registry/blob/1.3.x/app/src/test/java/io/apicurio/registry/RegistrySerdeTest.java
.
Omar,
Thank you for confirming the regression, and for preparing a fix.
Regards,
Mark
On Fri, 15 Jan 2021 at 09:23, Omar Al-Safi <om...@oalsafi.com> wrote:
> Hi Mark,
>
> indeed it looks like the additionalProperties option is broken in 3.7.0. I
> am pushing a fix for both branches, 3.7.x and master branch.
>
> Regards,
> Omar
>
>
> On Thu, Jan 14, 2021 at 6:41 PM Andrea Cosentino <an...@gmail.com>
> wrote:
>
> > I don't think it is feasible from our side to get to 3.6.0, it's up to
> you.
> >
> > Can you try to set the additionalProperties at component level like this
> > example?
> >
> >
> >
> https://github.com/apache/camel/blob/master/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaComponentTest.java#L63
> >
> > Il giorno gio 14 gen 2021 alle ore 17:28 mark <mp...@gmail.com> ha
> > scritto:
> >
> > > Hello,
> > > Until yesterday we were using Camel-K to send Avro-encoded messages to
> > > Kafka using code of the form,
> > >
> > > .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> > > +
> > >
> > >
> >
> "&keySerializerClass=org.apache.kafka.common.serialization.StringSerializer"
> > > +
> > > "&serializerClass=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> > > + "&additionalProperties.apicurio.registry.url=" +
> > > schemaRegistry
> > >
> > > Running this today, our Integration had picked up camel-kafka 3.7.0 and
> > > issue https://issues.apache.org/jira/browse/CAMEL-15770 which changed
> > the
> > > first two properties to keySerializer and valueSerializer. However
> > > adjusting the code to,
> > >
> > > .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> > > +
> > > "&keySerializer=org.apache.kafka.common.serialization.StringSerializer"
> > > +
> > > "&valueSerializer=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> > > + "&additionalProperties.apicurio.registry.url=" +
> schemaRegistry
> > >
> > > Leaves us no longer able to set additionalProperties. Attempts such as
> > > adding
> > >
> > > -d mvn:org.apache.camel/camel-kafka:[3.6.0]
> > >
> > > have no effect: we always get camel-kafka 3.7 and cannot pin the
> > > Integration to 3.6 or below. The relevant stack trace is below - please
> > can
> > > anyone help us either adjust the apicurio.registry.url for the new
> > library,
> > > or pin things back to 3.6 please?
> > >
> > > [1] Caused by: org.apache.kafka.common.KafkaException: Failed to
> > construct
> > > kafka producer
> > > [1] at
> > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:434)
> > > [1] at
> > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.component.kafka.KafkaProducer.doStart(KafkaProducer.java:115)
> > > [1] at
> > >
> org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.impl.engine.AbstractCamelContext.internalAddService(AbstractCamelContext.java:1425)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.impl.engine.AbstractCamelContext.addService(AbstractCamelContext.java:1343)
> > > [1] at
> > >
> org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:236)
> > > [1] at
> > >
> org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:101)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1487)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
> > > [1] ... 37 more
> > > [1] Caused by: java.lang.IllegalArgumentException: Missing registry
> base
> > > url, set apicurio.registry.url
> > > [1] at
> > >
> > >
> >
> io.apicurio.registry.utils.serde.AbstractKafkaSerDe.configure(AbstractKafkaSerDe.java:120)
> > > [1] at
> > >
> > >
> >
> io.apicurio.registry.utils.serde.AbstractKafkaStrategyAwareSerDe.configure(AbstractKafkaStrategyAwareSerDe.java:75)
> > > [1] at
> > >
> > >
> >
> io.apicurio.registry.utils.serde.AvroKafkaSerializer.configure(AvroKafkaSerializer.java:78)
> > > [1] at
> > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:369)
> > > [1] ... 49 more
> > >
> > > My dependencies are currently,
> > >
> > > // camel-k:
> > > dependency=mvn:org.apache.camel.quarkus/camel-quarkus-kafka:1.5.0
> > > // camel-k: dependency=mvn:org.apache.avro/avro:1.10.1
> > > // camel-k:
> dependency=mvn:org.glassfish.jersey.core/jersey-common:2.22.2
> > > // camel-k:
> > > dependency=mvn:io.apicurio/apicurio-registry-utils-serde:1.3.2.Final
> > >
> > > Many thanks in advance,
> > > Regards,
> > >
> > > Mark
> > >
> >
>
Re: Camel-K / Kafka / Avro: something broke overnight
Posted by Omar Al-Safi <om...@oalsafi.com>.
Hi Mark,
indeed it looks like the additionalProperties option is broken in 3.7.0. I
am pushing a fix for both branches, 3.7.x and master branch.
Regards,
Omar
On Thu, Jan 14, 2021 at 6:41 PM Andrea Cosentino <an...@gmail.com> wrote:
> I don't think it is feasible from our side to get to 3.6.0, it's up to you.
>
> Can you try to set the additionalProperties at component level like this
> example?
>
>
> https://github.com/apache/camel/blob/master/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaComponentTest.java#L63
>
> Il giorno gio 14 gen 2021 alle ore 17:28 mark <mp...@gmail.com> ha
> scritto:
>
> > Hello,
> > Until yesterday we were using Camel-K to send Avro-encoded messages to
> > Kafka using code of the form,
> >
> > .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> > +
> >
> >
> "&keySerializerClass=org.apache.kafka.common.serialization.StringSerializer"
> > +
> > "&serializerClass=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> > + "&additionalProperties.apicurio.registry.url=" +
> > schemaRegistry
> >
> > Running this today, our Integration had picked up camel-kafka 3.7.0 and
> > issue https://issues.apache.org/jira/browse/CAMEL-15770 which changed
> the
> > first two properties to keySerializer and valueSerializer. However
> > adjusting the code to,
> >
> > .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> > +
> > "&keySerializer=org.apache.kafka.common.serialization.StringSerializer"
> > +
> > "&valueSerializer=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> > + "&additionalProperties.apicurio.registry.url=" + schemaRegistry
> >
> > Leaves us no longer able to set additionalProperties. Attempts such as
> > adding
> >
> > -d mvn:org.apache.camel/camel-kafka:[3.6.0]
> >
> > have no effect: we always get camel-kafka 3.7 and cannot pin the
> > Integration to 3.6 or below. The relevant stack trace is below - please
> can
> > anyone help us either adjust the apicurio.registry.url for the new
> library,
> > or pin things back to 3.6 please?
> >
> > [1] Caused by: org.apache.kafka.common.KafkaException: Failed to
> construct
> > kafka producer
> > [1] at
> >
> >
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:434)
> > [1] at
> >
> >
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)
> > [1] at
> >
> >
> org.apache.camel.component.kafka.KafkaProducer.doStart(KafkaProducer.java:115)
> > [1] at
> > org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> > [1] at
> >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> > [1] at
> >
> >
> org.apache.camel.impl.engine.AbstractCamelContext.internalAddService(AbstractCamelContext.java:1425)
> > [1] at
> >
> >
> org.apache.camel.impl.engine.AbstractCamelContext.addService(AbstractCamelContext.java:1343)
> > [1] at
> > org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:236)
> > [1] at
> > org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> > [1] at
> >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> > [1] at
> >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:101)
> > [1] at
> >
> >
> org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1487)
> > [1] at
> >
> >
> org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
> > [1] ... 37 more
> > [1] Caused by: java.lang.IllegalArgumentException: Missing registry base
> > url, set apicurio.registry.url
> > [1] at
> >
> >
> io.apicurio.registry.utils.serde.AbstractKafkaSerDe.configure(AbstractKafkaSerDe.java:120)
> > [1] at
> >
> >
> io.apicurio.registry.utils.serde.AbstractKafkaStrategyAwareSerDe.configure(AbstractKafkaStrategyAwareSerDe.java:75)
> > [1] at
> >
> >
> io.apicurio.registry.utils.serde.AvroKafkaSerializer.configure(AvroKafkaSerializer.java:78)
> > [1] at
> >
> >
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:369)
> > [1] ... 49 more
> >
> > My dependencies are currently,
> >
> > // camel-k:
> > dependency=mvn:org.apache.camel.quarkus/camel-quarkus-kafka:1.5.0
> > // camel-k: dependency=mvn:org.apache.avro/avro:1.10.1
> > // camel-k: dependency=mvn:org.glassfish.jersey.core/jersey-common:2.22.2
> > // camel-k:
> > dependency=mvn:io.apicurio/apicurio-registry-utils-serde:1.3.2.Final
> >
> > Many thanks in advance,
> > Regards,
> >
> > Mark
> >
>
Re: Camel-K / Kafka / Avro: something broke overnight
Posted by Andrea Cosentino <an...@gmail.com>.
I don't think it is feasible from our side to get to 3.6.0, it's up to you.
Can you try to set the additionalProperties at component level like this
example?
https://github.com/apache/camel/blob/master/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaComponentTest.java#L63
Il giorno gio 14 gen 2021 alle ore 17:28 mark <mp...@gmail.com> ha
scritto:
> Hello,
> Until yesterday we were using Camel-K to send Avro-encoded messages to
> Kafka using code of the form,
>
> .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> +
>
> "&keySerializerClass=org.apache.kafka.common.serialization.StringSerializer"
> +
> "&serializerClass=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> + "&additionalProperties.apicurio.registry.url=" +
> schemaRegistry
>
> Running this today, our Integration had picked up camel-kafka 3.7.0 and
> issue https://issues.apache.org/jira/browse/CAMEL-15770 which changed the
> first two properties to keySerializer and valueSerializer. However
> adjusting the code to,
>
> .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> +
> "&keySerializer=org.apache.kafka.common.serialization.StringSerializer"
> +
> "&valueSerializer=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> + "&additionalProperties.apicurio.registry.url=" + schemaRegistry
>
> Leaves us no longer able to set additionalProperties. Attempts such as
> adding
>
> -d mvn:org.apache.camel/camel-kafka:[3.6.0]
>
> have no effect: we always get camel-kafka 3.7 and cannot pin the
> Integration to 3.6 or below. The relevant stack trace is below - please can
> anyone help us either adjust the apicurio.registry.url for the new library,
> or pin things back to 3.6 please?
>
> [1] Caused by: org.apache.kafka.common.KafkaException: Failed to construct
> kafka producer
> [1] at
>
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:434)
> [1] at
>
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)
> [1] at
>
> org.apache.camel.component.kafka.KafkaProducer.doStart(KafkaProducer.java:115)
> [1] at
> org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> [1] at
>
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> [1] at
>
> org.apache.camel.impl.engine.AbstractCamelContext.internalAddService(AbstractCamelContext.java:1425)
> [1] at
>
> org.apache.camel.impl.engine.AbstractCamelContext.addService(AbstractCamelContext.java:1343)
> [1] at
> org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:236)
> [1] at
> org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> [1] at
>
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> [1] at
>
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:101)
> [1] at
>
> org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1487)
> [1] at
>
> org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
> [1] ... 37 more
> [1] Caused by: java.lang.IllegalArgumentException: Missing registry base
> url, set apicurio.registry.url
> [1] at
>
> io.apicurio.registry.utils.serde.AbstractKafkaSerDe.configure(AbstractKafkaSerDe.java:120)
> [1] at
>
> io.apicurio.registry.utils.serde.AbstractKafkaStrategyAwareSerDe.configure(AbstractKafkaStrategyAwareSerDe.java:75)
> [1] at
>
> io.apicurio.registry.utils.serde.AvroKafkaSerializer.configure(AvroKafkaSerializer.java:78)
> [1] at
>
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:369)
> [1] ... 49 more
>
> My dependencies are currently,
>
> // camel-k:
> dependency=mvn:org.apache.camel.quarkus/camel-quarkus-kafka:1.5.0
> // camel-k: dependency=mvn:org.apache.avro/avro:1.10.1
> // camel-k: dependency=mvn:org.glassfish.jersey.core/jersey-common:2.22.2
> // camel-k:
> dependency=mvn:io.apicurio/apicurio-registry-utils-serde:1.3.2.Final
>
> Many thanks in advance,
> Regards,
>
> Mark
>