You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Ioannis Polyzos <i....@gmail.com> on 2014/03/30 01:43:09 UTC
Camel-Kafka Component
Hello all,
I was recently working on a *camel-kafka* component that I would like to
contribute back to the Apache Camel project.
The component allows to configure the majority of the kafka client
parameters and to transfer both the body and the exchange.It definitely
need more work and therefore I would like to kindly ask for your feedback
on this early work.
* P.S You could find the source code
at https://github.com/ipolyzos/camel-kafka/
<https://github.com/ipolyzos/camel-kafka/>*
thank you and regards,
Ioannis
Re: Camel-Kafka Component
Posted by Ioannis Polyzos <i....@gmail.com>.
Hi all,
Based on our earlier discussion I have gone through and submitted a patch
for the camel-kafka component.
The patch has been submitted in CAMEL-7339 (
https://issues.apache.org/jira/browse/CAMEL-7339) and I would like to
kindly ask your feedback on this.
Among the the changes the patch introduce is in the syntax pattern of
kafka component. The current component implementation syntax is :
* kafka:<SERVERs>?topic=<TOPIC>*
while in the proposed implementation is
kafka:<TOPIC??metadataBrokerList=<SERVERs>
On Sun, Mar 30, 2014 at 10:05 AM, Ioannis Polyzos <i....@gmail.com>wrote:
> Hi Richard,
>
> Thanks for your kind response, I will proceed adding documentation as for
> CAMEL-7288 as well.
>
> Thanks
>
>
> On Sun, Mar 30, 2014 at 9:18 AM, Richard Kettelerij <
> richardkettelerij@gmail.com> wrote:
>
>> As Claus mentioned it would be great if you could improve the recently
>> added camel-kafka component (wherever possible).
>>
>> The new Kafka component also missing documentation (see
>> https://issues.apache.org/jira/browse/CAMEL-7288), that would be also be
>> much appreciated.
>>
>> Regards,
>> Richard
>> http://richardlog.com
>>
>>
>> On Sun, Mar 30, 2014 at 1:43 AM, Ioannis Polyzos <i.polyzos@gmail.com
>> >wrote:
>>
>> > Hello all,
>> >
>> > I was recently working on a *camel-kafka* component that I would like to
>> > contribute back to the Apache Camel project.
>> >
>> > The component allows to configure the majority of the kafka client
>> > parameters and to transfer both the body and the exchange.It definitely
>> > need more work and therefore I would like to kindly ask for your
>> feedback
>> > on this early work.
>> >
>> > * P.S You could find the source code
>> > at https://github.com/ipolyzos/camel-kafka/
>> > <https://github.com/ipolyzos/camel-kafka/>*
>> >
>> > thank you and regards,
>> > Ioannis
>> >
>>
>
>
Re: Camel-Kafka Component
Posted by mekki20 <ch...@gmx.de>.
Thanks for the answer. On the Apache Camel page I see for both releases
2.16.0 and 2.15.3 a planned release date of Q3 2015. Do you have already a
more precise date for any of the two?
Regards,
Christian
Mark Mindenhall [via Camel] <ml...@n5.nabble.com>
schrieb am Di., 21. Juli 2015 um 03:13 Uhr:
> Hi Christian,
>
> The patch Claus referenced was also applied to the 2.15.x branch, so it
> will be part of the 2.15.3 release as well (if you’d prefer to avoid using
> snapshot builds).
>
> Best,
> Mark
>
>
> On July 20, 2015 at 6:02:47 AM, Claus Ibsen ([hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5769637&i=0><mailto:[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=5769637&i=1>>) wrote:
>
> Hi
>
> The upcoming Camel 2.16 should support different encoders so you can
> use binary etc. You are welcome to try 2.16-SNAPSHOT to give it a test
> spin.
>
> The commit is
>
> https://github.com/apache/camel/commit/1af2133f68498bd91361a2ca2237fb2426233721
>
> On Mon, Jul 20, 2015 at 1:58 PM, mekki20 <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5769637&i=2>> wrote:
>
> > Hello,
> >
> > I've started to use the Camel-Kafka Component but also wanted to send
> binary
> > messages, not only String encoded ones. Attached is the changed
> > KafakProducer.java file.
> > I'm happy to get your feedback and how to proceed.
> >
> > Regards,
> > Christian
> >
> >
> >
> > --
> > View this message in context:
> http://camel.465427.n5.nabble.com/Camel-Kafka-Component-tp5749525p5769561.html
> > Sent from the Camel Development mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2nd edition: http://www.manning.com/ibsen2
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/Camel-Kafka-Component-tp5749525p5769637.html
> To unsubscribe from Camel-Kafka Component, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5749525&code=Y2hyaXN0aWFuX3JhcHBAZ214LmRlfDU3NDk1MjV8LTE4MzYxNjI3MDA=>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Kafka-Component-tp5749525p5769726.html
Sent from the Camel Development mailing list archive at Nabble.com.
Re: Camel-Kafka Component
Posted by Mark Mindenhall <ma...@machineshop.io>.
Hi Christian,
The patch Claus referenced was also applied to the 2.15.x branch, so it will be part of the 2.15.3 release as well (if you’d prefer to avoid using snapshot builds).
Best,
Mark
On July 20, 2015 at 6:02:47 AM, Claus Ibsen (claus.ibsen@gmail.com<ma...@gmail.com>) wrote:
Hi
The upcoming Camel 2.16 should support different encoders so you can
use binary etc. You are welcome to try 2.16-SNAPSHOT to give it a test
spin.
The commit is
https://github.com/apache/camel/commit/1af2133f68498bd91361a2ca2237fb2426233721
On Mon, Jul 20, 2015 at 1:58 PM, mekki20 <ch...@gmx.de> wrote:
> Hello,
>
> I've started to use the Camel-Kafka Component but also wanted to send binary
> messages, not only String encoded ones. Attached is the changed
> KafakProducer.java file.
> I'm happy to get your feedback and how to proceed.
>
> Regards,
> Christian
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-Kafka-Component-tp5749525p5769561.html
> Sent from the Camel Development mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2nd edition: http://www.manning.com/ibsen2
Re: Camel-Kafka Component
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
The upcoming Camel 2.16 should support different encoders so you can
use binary etc. You are welcome to try 2.16-SNAPSHOT to give it a test
spin.
The commit is
https://github.com/apache/camel/commit/1af2133f68498bd91361a2ca2237fb2426233721
On Mon, Jul 20, 2015 at 1:58 PM, mekki20 <ch...@gmx.de> wrote:
> Hello,
>
> I've started to use the Camel-Kafka Component but also wanted to send binary
> messages, not only String encoded ones. Attached is the changed
> KafakProducer.java file.
> I'm happy to get your feedback and how to proceed.
>
> Regards,
> Christian
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-Kafka-Component-tp5749525p5769561.html
> Sent from the Camel Development mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2nd edition: http://www.manning.com/ibsen2
Re: Camel-Kafka Component
Posted by mekki20 <ch...@gmx.de>.
Hello,
I've started to use the Camel-Kafka Component but also wanted to send binary
messages, not only String encoded ones. Attached is the changed
KafakProducer.java file.
I'm happy to get your feedback and how to proceed.
Regards,
Christian
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Kafka-Component-tp5749525p5769561.html
Sent from the Camel Development mailing list archive at Nabble.com.
Re: Camel-Kafka Component
Posted by Ioannis Polyzos <i....@gmail.com>.
Hi all,
Based on our earlier discussion I have gone through and submitted a patch
for the camel-kafka component.
The patch has been submitted in CAMEL-7339 (
https://issues.apache.org/jira/browse/CAMEL-7339) and I would like to
kindly ask your feedback on this.
Among the the changes the patch introduce is in the syntax pattern of
kafka component. The current component implementation syntax is :
* kafka:<SERVERs>?topic=<TOPIC>*
while in the proposed implementation is
*kafka:<TOPIC??metadataBrokerList=<SERVERs> *
Another change is that by default data are not in the form of Strings*(i.e
KeyedMessage<String, String>) *but in the form of byte arrays *(i.e
KeyedMessage<String,
byte[]>)*. And finally allow for the transfer of the whole exchange by
using the *DefaultExchangeHolder*.
Cheers,
Ioannis
On Sun, Mar 30, 2014 at 10:05 AM, Ioannis Polyzos <i....@gmail.com>wrote:
> Hi Richard,
>
> Thanks for your kind response, I will proceed adding documentation as for
> CAMEL-7288 as well.
>
> Thanks
>
>
> On Sun, Mar 30, 2014 at 9:18 AM, Richard Kettelerij <
> richardkettelerij@gmail.com> wrote:
>
>> As Claus mentioned it would be great if you could improve the recently
>> added camel-kafka component (wherever possible).
>>
>> The new Kafka component also missing documentation (see
>> https://issues.apache.org/jira/browse/CAMEL-7288), that would be also be
>> much appreciated.
>>
>> Regards,
>> Richard
>> http://richardlog.com
>>
>>
>> On Sun, Mar 30, 2014 at 1:43 AM, Ioannis Polyzos <i.polyzos@gmail.com
>> >wrote:
>>
>> > Hello all,
>> >
>> > I was recently working on a *camel-kafka* component that I would like to
>> > contribute back to the Apache Camel project.
>> >
>> > The component allows to configure the majority of the kafka client
>> > parameters and to transfer both the body and the exchange.It definitely
>> > need more work and therefore I would like to kindly ask for your
>> feedback
>> > on this early work.
>> >
>> > * P.S You could find the source code
>> > at https://github.com/ipolyzos/camel-kafka/
>> > <https://github.com/ipolyzos/camel-kafka/>*
>> >
>> > thank you and regards,
>> > Ioannis
>> >
>>
>
>
Re: Camel-Kafka Component
Posted by Ioannis Polyzos <i....@gmail.com>.
Hi Richard,
Thanks for your kind response, I will proceed adding documentation as for
CAMEL-7288 as well.
Thanks
On Sun, Mar 30, 2014 at 9:18 AM, Richard Kettelerij <
richardkettelerij@gmail.com> wrote:
> As Claus mentioned it would be great if you could improve the recently
> added camel-kafka component (wherever possible).
>
> The new Kafka component also missing documentation (see
> https://issues.apache.org/jira/browse/CAMEL-7288), that would be also be
> much appreciated.
>
> Regards,
> Richard
> http://richardlog.com
>
>
> On Sun, Mar 30, 2014 at 1:43 AM, Ioannis Polyzos <i.polyzos@gmail.com
> >wrote:
>
> > Hello all,
> >
> > I was recently working on a *camel-kafka* component that I would like to
> > contribute back to the Apache Camel project.
> >
> > The component allows to configure the majority of the kafka client
> > parameters and to transfer both the body and the exchange.It definitely
> > need more work and therefore I would like to kindly ask for your feedback
> > on this early work.
> >
> > * P.S You could find the source code
> > at https://github.com/ipolyzos/camel-kafka/
> > <https://github.com/ipolyzos/camel-kafka/>*
> >
> > thank you and regards,
> > Ioannis
> >
>
Re: Camel-Kafka Component
Posted by Richard Kettelerij <ri...@gmail.com>.
As Claus mentioned it would be great if you could improve the recently
added camel-kafka component (wherever possible).
The new Kafka component also missing documentation (see
https://issues.apache.org/jira/browse/CAMEL-7288), that would be also be
much appreciated.
Regards,
Richard
http://richardlog.com
On Sun, Mar 30, 2014 at 1:43 AM, Ioannis Polyzos <i....@gmail.com>wrote:
> Hello all,
>
> I was recently working on a *camel-kafka* component that I would like to
> contribute back to the Apache Camel project.
>
> The component allows to configure the majority of the kafka client
> parameters and to transfer both the body and the exchange.It definitely
> need more work and therefore I would like to kindly ask for your feedback
> on this early work.
>
> * P.S You could find the source code
> at https://github.com/ipolyzos/camel-kafka/
> <https://github.com/ipolyzos/camel-kafka/>*
>
> thank you and regards,
> Ioannis
>
Re: Camel-Kafka Component
Posted by Ioannis Polyzos <i....@gmail.com>.
Hi Claus,
I would be more than happy to proceed with so.
I will go through and proceed with the relevant changes in the next days.
thanks
On Sun, Mar 30, 2014 at 7:57 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> We recently added a new camel-kafka component which was released in
> Camel 2.13.0.
> https://github.com/apache/camel/tree/master/components/camel-kafka
>
> You are welcome to help improve the existing component.
>
>
> On Sun, Mar 30, 2014 at 1:43 AM, Ioannis Polyzos <i....@gmail.com>
> wrote:
> > Hello all,
> >
> > I was recently working on a *camel-kafka* component that I would like to
> > contribute back to the Apache Camel project.
> >
> > The component allows to configure the majority of the kafka client
> > parameters and to transfer both the body and the exchange.It definitely
> > need more work and therefore I would like to kindly ask for your feedback
> > on this early work.
> >
> > * P.S You could find the source code
> > at https://github.com/ipolyzos/camel-kafka/
> > <https://github.com/ipolyzos/camel-kafka/>*
> >
> > thank you and regards,
> > Ioannis
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io
>
Re: Camel-Kafka Component
Posted by sat <sa...@yahoo.com>.
I am answering my own question -
Looks like we have to explicitly specify the serializerClass for the
producer like this
from("direct:testingKafka").to("kafka:localhost:9092?topic=test&serializerClass=kafka.serializer.StringEncoder")
This worked.
Would be nice if it is handled in Camel if possible. I don't know enough
about it or Kafka at the moment to submit a patch. Would like feedback and
I'll see if I can bake it into a pull request.
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Kafka-Component-tp5749525p5756455.html
Sent from the Camel Development mailing list archive at Nabble.com.
Re: Camel-Kafka Component
Posted by sat <sa...@yahoo.com>.
I've tried to give the kafka component a whirl (disclaimer - I am new to both
camel and kafka).
Tried this and got this exception when sending a message to kafka
In my spring class that produces:
@Autowired
private ProducerTemplate template;
public void sendKakfaMessage(Integer id){
template.sendBodyAndHeader("direct:testingKafka",
String.valueOf(id), KafkaConstants.PARTITION_KEY, "1");
}
In my route in a configuration class
@Bean
public RouteBuilder routeBuilder() {
return new RouteBuilder() {
@Override
public void configure() throws Exception {
from("direct:testingKafka").to("kafka:localhost:9092?topic=test");
}
};
}
Gives me the below exception.
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
[B
at kafka.serializer.DefaultEncoder.toBytes(Encoder.scala:34)
at
kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:128)
at
kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:125)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at
kafka.producer.async.DefaultEventHandler.serialize(DefaultEventHandler.scala:125)
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:52)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at
org.apache.camel.component.kafka.KafkaProducer.process(KafkaProducer.java:78)
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:113)
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Kafka-Component-tp5749525p5756454.html
Sent from the Camel Development mailing list archive at Nabble.com.
Re: Camel-Kafka Component
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
We recently added a new camel-kafka component which was released in
Camel 2.13.0.
https://github.com/apache/camel/tree/master/components/camel-kafka
You are welcome to help improve the existing component.
On Sun, Mar 30, 2014 at 1:43 AM, Ioannis Polyzos <i....@gmail.com> wrote:
> Hello all,
>
> I was recently working on a *camel-kafka* component that I would like to
> contribute back to the Apache Camel project.
>
> The component allows to configure the majority of the kafka client
> parameters and to transfer both the body and the exchange.It definitely
> need more work and therefore I would like to kindly ask for your feedback
> on this early work.
>
> * P.S You could find the source code
> at https://github.com/ipolyzos/camel-kafka/
> <https://github.com/ipolyzos/camel-kafka/>*
>
> thank you and regards,
> Ioannis
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io