You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Andrea Spina <an...@radicalbit.io> on 2018/08/23 13:44:58 UTC
Kafka Connect REST connector with additional logging to kafka
Dear community,
I'd figure out a kafka connect HTTP/REST sink connector X capable of
achieving PUT actions on an external web service A. I saw some side
projects already implementing something similar [1]. In addition to this
feature, I'd like to log to kafka the answers returning by A so that I can
use it later somewhere else.
Since I can't find some examples on that (connectors logging to kafka as
"side effect"), my first question is if the former design is something
doable by Kafka Connect and if it well fits the architecture.
I need this connector because I'm trying to keep it stateless (avoid
storing any state), because AFAIU Kafka Connect technology is not demanded
to keep states along its implementation. Theoretically, if I could store a
state within my kafka connector instance, it'd be way easier achieving my
task.
So my last question is: is, the fact that connectors should be stateless,
actually true? And why shouldn't I store states through a connector? (I can
eventually open another thread on this if this one is overloaded).
Any help is much appreciated. Thank you for your work.
[1] - https://github.com/llofberg/kafka-connect-rest
--
*Andrea Spina*
Software Engineer @ Radicalbit Srl
Via Borsieri 41, 20159, Milano - IT
Re: Kafka Connect REST connector with additional logging to kafka
Posted by Andrea Spina <an...@radicalbit.io>.
Hello community, any hints on this, then?
Thank you, Andrea
On Thu, Aug 23, 2018, 16:15 Andrea Spina <an...@radicalbit.io> wrote:
> Hi Andrew,
> Thank you for your answer. Sadly I'm not allowed to add this module to my
> technologies by now.
>
> 2018-08-23 16:00 GMT+02:00 Andrew Otto <ot...@wikimedia.org>:
>
>> Hiya, this doesn’t help answer your question, but as an FYI, Wikimedia has
>> implemented https://github.com/wikimedia/change-propagation to do what
>> you
>> are trying to do: issue HTTP requests (and other things) triggered by
>> incoming messages in Kafka.
>>
>> On Thu, Aug 23, 2018 at 9:45 AM Andrea Spina <an...@radicalbit.io>
>> wrote:
>>
>> > Dear community,
>> >
>> > I'd figure out a kafka connect HTTP/REST sink connector X capable of
>> > achieving PUT actions on an external web service A. I saw some side
>> > projects already implementing something similar [1]. In addition to this
>> > feature, I'd like to log to kafka the answers returning by A so that I
>> can
>> > use it later somewhere else.
>> >
>> > Since I can't find some examples on that (connectors logging to kafka as
>> > "side effect"), my first question is if the former design is something
>> > doable by Kafka Connect and if it well fits the architecture.
>> >
>> > I need this connector because I'm trying to keep it stateless (avoid
>> > storing any state), because AFAIU Kafka Connect technology is not
>> demanded
>> > to keep states along its implementation. Theoretically, if I could
>> store a
>> > state within my kafka connector instance, it'd be way easier achieving
>> my
>> > task.
>> >
>> > So my last question is: is, the fact that connectors should be
>> stateless,
>> > actually true? And why shouldn't I store states through a connector? (I
>> can
>> > eventually open another thread on this if this one is overloaded).
>> >
>> > Any help is much appreciated. Thank you for your work.
>> >
>> >
>> > [1] - https://github.com/llofberg/kafka-connect-rest
>> > --
>> > *Andrea Spina*
>> > Software Engineer @ Radicalbit Srl
>> > Via Borsieri 41, 20159, Milano - IT
>> >
>>
>
>
>
> --
> *Andrea Spina*
> Software Engineer @ Radicalbit Srl
> Via Borsieri 41, 20159, Milano - IT
>
Re: Kafka Connect REST connector with additional logging to kafka
Posted by Andrea Spina <an...@radicalbit.io>.
Hi Andrew,
Thank you for your answer. Sadly I'm not allowed to add this module to my
technologies by now.
2018-08-23 16:00 GMT+02:00 Andrew Otto <ot...@wikimedia.org>:
> Hiya, this doesn’t help answer your question, but as an FYI, Wikimedia has
> implemented https://github.com/wikimedia/change-propagation to do what you
> are trying to do: issue HTTP requests (and other things) triggered by
> incoming messages in Kafka.
>
> On Thu, Aug 23, 2018 at 9:45 AM Andrea Spina <an...@radicalbit.io>
> wrote:
>
> > Dear community,
> >
> > I'd figure out a kafka connect HTTP/REST sink connector X capable of
> > achieving PUT actions on an external web service A. I saw some side
> > projects already implementing something similar [1]. In addition to this
> > feature, I'd like to log to kafka the answers returning by A so that I
> can
> > use it later somewhere else.
> >
> > Since I can't find some examples on that (connectors logging to kafka as
> > "side effect"), my first question is if the former design is something
> > doable by Kafka Connect and if it well fits the architecture.
> >
> > I need this connector because I'm trying to keep it stateless (avoid
> > storing any state), because AFAIU Kafka Connect technology is not
> demanded
> > to keep states along its implementation. Theoretically, if I could store
> a
> > state within my kafka connector instance, it'd be way easier achieving my
> > task.
> >
> > So my last question is: is, the fact that connectors should be stateless,
> > actually true? And why shouldn't I store states through a connector? (I
> can
> > eventually open another thread on this if this one is overloaded).
> >
> > Any help is much appreciated. Thank you for your work.
> >
> >
> > [1] - https://github.com/llofberg/kafka-connect-rest
> > --
> > *Andrea Spina*
> > Software Engineer @ Radicalbit Srl
> > Via Borsieri 41, 20159, Milano - IT
> >
>
--
*Andrea Spina*
Software Engineer @ Radicalbit Srl
Via Borsieri 41, 20159, Milano - IT
Re: Kafka Connect REST connector with additional logging to kafka
Posted by Andrew Otto <ot...@wikimedia.org>.
Hiya, this doesn’t help answer your question, but as an FYI, Wikimedia has
implemented https://github.com/wikimedia/change-propagation to do what you
are trying to do: issue HTTP requests (and other things) triggered by
incoming messages in Kafka.
On Thu, Aug 23, 2018 at 9:45 AM Andrea Spina <an...@radicalbit.io>
wrote:
> Dear community,
>
> I'd figure out a kafka connect HTTP/REST sink connector X capable of
> achieving PUT actions on an external web service A. I saw some side
> projects already implementing something similar [1]. In addition to this
> feature, I'd like to log to kafka the answers returning by A so that I can
> use it later somewhere else.
>
> Since I can't find some examples on that (connectors logging to kafka as
> "side effect"), my first question is if the former design is something
> doable by Kafka Connect and if it well fits the architecture.
>
> I need this connector because I'm trying to keep it stateless (avoid
> storing any state), because AFAIU Kafka Connect technology is not demanded
> to keep states along its implementation. Theoretically, if I could store a
> state within my kafka connector instance, it'd be way easier achieving my
> task.
>
> So my last question is: is, the fact that connectors should be stateless,
> actually true? And why shouldn't I store states through a connector? (I can
> eventually open another thread on this if this one is overloaded).
>
> Any help is much appreciated. Thank you for your work.
>
>
> [1] - https://github.com/llofberg/kafka-connect-rest
> --
> *Andrea Spina*
> Software Engineer @ Radicalbit Srl
> Via Borsieri 41, 20159, Milano - IT
>