You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Michael André Pearce <mi...@me.com> on 2017/10/06 06:48:56 UTC
Kafka ServiceConnector
Hi All,
I am looking to contribute an artemis ServiceConnector that would leverage the existing service connector interface to bridge from a queue into a kafka topic similar to bridge between artemis clusters.
Idea is to make it configurable how to transform and encode the message in kafka but as amqp is becoming very well supported by artemis provide out the box that the message be encoded in amqp byte protocol.
It something I’ve been working on privately for a bit, but since becoming a committer here is really like to share it.
I hope if I get a positive response here to get a branch ready soon, but feedback if this is welcomed would be great.
Cheers
Mike
Sent from my iPhone
Re: Kafka ServiceConnector
Posted by Michael André Pearce <mi...@me.com>.
Im 0.5 this idea atm as there isn’t such a page.
So I can become more sold on this to become a +1, being a committer how would I go about creating such an ecosystem page within ActiveMQ Artemis site.
Cheers
Mike
Sent from my iPhone
> On 27 Oct 2017, at 11:05, Martyn Taylor <mt...@redhat.com> wrote:
>
> Looking at the Kafka Connect example. I see that the integration
> components are all maintained outside of the Kafka project. They're listed
> here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>
> How about we do something similar and create a wiki/web page linking the
> integration projects. I realise our website isn't the most accessible atm
> but going forward this should be improved.
>
> On Thu, Oct 26, 2017 at 5:42 PM, Robbie Gemmell <ro...@gmail.com>
> wrote:
>
>> The related PR is https://github.com/apache/activemq-artemis/pull/1607
>> for anyone not following along thus far, or later reference.
>>
>> My main view there was that such an extension would be better
>> maintained seperate from the main broker repository+distribution for a
>> variety of reasons. After that I didn't have a strong view on where it
>> lives, but I would say that I don't actually see any issue at all with
>> it being outwith the ActiveMQ project. To me thats just a key thing
>> enabled by such plugin points existing, and not uncommon with other
>> projects/components.
>>
>> Robbie
>>
>> On 26 October 2017 at 16:46, Michael André Pearce
>> <mi...@me.com> wrote:
>>> As noted on the PR part of the PR discussion is should Service
>> Connectors exist with the broker code.
>>>
>>> Service Connectors / Extensions are ActiveMQ Artemis specific.
>>>
>>> I think having them within the project space is good so we can grow an
>> eco system of some, people can just use will aid in having better broker
>> adoption as tools people look for would be available if contributed.
>>>
>>> As stated above it seems the general consensus on the PR is though liked
>> is the broker code base best for them to live?
>>>
>>> If we don’t wish them to exist with broker code so they can have their
>> own release cycle, and don’t murky the water of the broker, I would like to
>> discuss if we could make a sub project for these to live, and possibly
>> propose a sub project be setup called “ActiveMQ Artemis Integrations” ?
>>>
>>> If setup I would also propose maybe moving the spring integration there
>> also.
>>>
>>> Cheers
>>> Mike
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Sent from my iPhone
>>>
>>>> On 6 Oct 2017, at 12:20, Michael André Pearce <
>> michael.andre.pearce@me.com> wrote:
>>>>
>>>> Yes and no, it would be same as arguing that core bridge between
>> artemis clusters could also be separated components.
>>>>
>>>> The idea is to effectively have similar to a core bridge but to kafka.
>>>>
>>>> Some key parts why are (and probably similar to why core bridge is
>> implemented within the broker)
>>>>
>>>> Latency
>>>> Extra processes and hops
>>>> Out the box in broker support
>>>>
>>>> Cheers
>>>> Mike
>>>>
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 6 Oct 2017, at 12:14, Andy Taylor <an...@gmail.com> wrote:
>>>>>
>>>>> I think https://github.com/ppatierno/kafka-connect-amqp is an example
>> of
>>>>> this
>>>>>
>>>>> On 6 October 2017 at 12:07, Christopher Shannon <
>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>
>>>>>> Why not use Kafka's Connect framework? Seems like building a source
>>>>>> connector here would make the most sense. This is not to say I'm
>> against
>>>>>> the idea I'm just wondering the benefits of doing it from the Artemis
>> side
>>>>>> of things when Kafka already has a framework that can be used to
>>>>>> import/export data.
>>>>>>
>>>>>> On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce <
>>>>>> michael.andre.pearce@me.com> wrote:
>>>>>>
>>>>>>> Hi Andy,
>>>>>>>
>>>>>>> As you may be aware or not we run both activemq and kafka each has a
>>>>>>> specific plus point to it, use the right tool for the right job.
>>>>>>>
>>>>>>> We use activemq as our feature rich, fully jms compliant and
>> performant
>>>>>>> broker.
>>>>>>>
>>>>>>> And use kafka more for post processing and data streaming, with
>>>>>> historical
>>>>>>> feed also.
>>>>>>>
>>>>>>> Idea here is that producers and consumers bind into our activemq
>> where
>>>>>> jms
>>>>>>> api is wanted or needed, lower per message latencies etc. but then
>> we can
>>>>>>> bridge messages from a core queue into kafka so that the data can
>> flow to
>>>>>>> it as fast as possible and consumed from there also.
>>>>>>>
>>>>>>> Idea is to try make a hybrid taking best of both.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Mike
>>>>>>>
>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>>> On 6 Oct 2017, at 09:37, Andy Taylor <an...@gmail.com>
>> wrote:
>>>>>>>>
>>>>>>>> Whats the use case here Michael, sounds like a cool idea and
>> something
>>>>>>> that
>>>>>>>> has been mentioned by different people but I am yet to understand
>> how
>>>>>>> this
>>>>>>>> could be used.
>>>>>>>>
>>>>>>>> On 6 October 2017 at 07:48, Michael André Pearce <
>>>>>>>> michael.andre.pearce@me.com> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> I am looking to contribute an artemis ServiceConnector that would
>>>>>>> leverage
>>>>>>>>> the existing service connector interface to bridge from a queue
>> into a
>>>>>>>>> kafka topic similar to bridge between artemis clusters.
>>>>>>>>>
>>>>>>>>> Idea is to make it configurable how to transform and encode the
>>>>>> message
>>>>>>> in
>>>>>>>>> kafka but as amqp is becoming very well supported by artemis
>> provide
>>>>>> out
>>>>>>>>> the box that the message be encoded in amqp byte protocol.
>>>>>>>>>
>>>>>>>>> It something I’ve been working on privately for a bit, but since
>>>>>>> becoming
>>>>>>>>> a committer here is really like to share it.
>>>>>>>>>
>>>>>>>>> I hope if I get a positive response here to get a branch ready
>> soon,
>>>>>> but
>>>>>>>>> feedback if this is welcomed would be great.
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>
>>
Re: Kafka ServiceConnector
Posted by Justin Bertram <jb...@apache.org>.
In my opinion integrations like this are best kept outside the broker. If
they use the service connector hooks then there won't be any performance
impact whether or not they are in the broker code-base.
Also, I'm not sure I understand the comparison made between this
integration and the core bridge. The core bridge is a necessary bit of
functionality to enable clustering. Clustering is a core broker feature.
Integration with Kafka (while nice) is not a core broker feature.
As long as they aren't in the Artemis code-base then I don't think it's a
big deal where the integrations live as long as there some up-to-date
documentation about where to find them.
Justin
On Fri, Oct 27, 2017 at 5:05 AM, Martyn Taylor <mt...@redhat.com> wrote:
> Looking at the Kafka Connect example. I see that the integration
> components are all maintained outside of the Kafka project. They're listed
> here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>
> How about we do something similar and create a wiki/web page linking the
> integration projects. I realise our website isn't the most accessible atm
> but going forward this should be improved.
>
> On Thu, Oct 26, 2017 at 5:42 PM, Robbie Gemmell <ro...@gmail.com>
> wrote:
>
> > The related PR is https://github.com/apache/activemq-artemis/pull/1607
> > for anyone not following along thus far, or later reference.
> >
> > My main view there was that such an extension would be better
> > maintained seperate from the main broker repository+distribution for a
> > variety of reasons. After that I didn't have a strong view on where it
> > lives, but I would say that I don't actually see any issue at all with
> > it being outwith the ActiveMQ project. To me thats just a key thing
> > enabled by such plugin points existing, and not uncommon with other
> > projects/components.
> >
> > Robbie
> >
> > On 26 October 2017 at 16:46, Michael André Pearce
> > <mi...@me.com> wrote:
> > > As noted on the PR part of the PR discussion is should Service
> > Connectors exist with the broker code.
> > >
> > > Service Connectors / Extensions are ActiveMQ Artemis specific.
> > >
> > > I think having them within the project space is good so we can grow an
> > eco system of some, people can just use will aid in having better broker
> > adoption as tools people look for would be available if contributed.
> > >
> > > As stated above it seems the general consensus on the PR is though
> liked
> > is the broker code base best for them to live?
> > >
> > > If we don’t wish them to exist with broker code so they can have their
> > own release cycle, and don’t murky the water of the broker, I would like
> to
> > discuss if we could make a sub project for these to live, and possibly
> > propose a sub project be setup called “ActiveMQ Artemis Integrations” ?
> > >
> > > If setup I would also propose maybe moving the spring integration there
> > also.
> > >
> > > Cheers
> > > Mike
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Sent from my iPhone
> > >
> > >> On 6 Oct 2017, at 12:20, Michael André Pearce <
> > michael.andre.pearce@me.com> wrote:
> > >>
> > >> Yes and no, it would be same as arguing that core bridge between
> > artemis clusters could also be separated components.
> > >>
> > >> The idea is to effectively have similar to a core bridge but to kafka.
> > >>
> > >> Some key parts why are (and probably similar to why core bridge is
> > implemented within the broker)
> > >>
> > >> Latency
> > >> Extra processes and hops
> > >> Out the box in broker support
> > >>
> > >> Cheers
> > >> Mike
> > >>
> > >>
> > >> Sent from my iPhone
> > >>
> > >>> On 6 Oct 2017, at 12:14, Andy Taylor <an...@gmail.com> wrote:
> > >>>
> > >>> I think https://github.com/ppatierno/kafka-connect-amqp is an
> example
> > of
> > >>> this
> > >>>
> > >>> On 6 October 2017 at 12:07, Christopher Shannon <
> > >>> christopher.l.shannon@gmail.com> wrote:
> > >>>
> > >>>> Why not use Kafka's Connect framework? Seems like building a source
> > >>>> connector here would make the most sense. This is not to say I'm
> > against
> > >>>> the idea I'm just wondering the benefits of doing it from the
> Artemis
> > side
> > >>>> of things when Kafka already has a framework that can be used to
> > >>>> import/export data.
> > >>>>
> > >>>> On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce <
> > >>>> michael.andre.pearce@me.com> wrote:
> > >>>>
> > >>>>> Hi Andy,
> > >>>>>
> > >>>>> As you may be aware or not we run both activemq and kafka each has
> a
> > >>>>> specific plus point to it, use the right tool for the right job.
> > >>>>>
> > >>>>> We use activemq as our feature rich, fully jms compliant and
> > performant
> > >>>>> broker.
> > >>>>>
> > >>>>> And use kafka more for post processing and data streaming, with
> > >>>> historical
> > >>>>> feed also.
> > >>>>>
> > >>>>> Idea here is that producers and consumers bind into our activemq
> > where
> > >>>> jms
> > >>>>> api is wanted or needed, lower per message latencies etc. but then
> > we can
> > >>>>> bridge messages from a core queue into kafka so that the data can
> > flow to
> > >>>>> it as fast as possible and consumed from there also.
> > >>>>>
> > >>>>> Idea is to try make a hybrid taking best of both.
> > >>>>>
> > >>>>> Cheers
> > >>>>> Mike
> > >>>>>
> > >>>>> Sent from my iPhone
> > >>>>>
> > >>>>>> On 6 Oct 2017, at 09:37, Andy Taylor <an...@gmail.com>
> > wrote:
> > >>>>>>
> > >>>>>> Whats the use case here Michael, sounds like a cool idea and
> > something
> > >>>>> that
> > >>>>>> has been mentioned by different people but I am yet to understand
> > how
> > >>>>> this
> > >>>>>> could be used.
> > >>>>>>
> > >>>>>> On 6 October 2017 at 07:48, Michael André Pearce <
> > >>>>>> michael.andre.pearce@me.com> wrote:
> > >>>>>>
> > >>>>>>> Hi All,
> > >>>>>>>
> > >>>>>>> I am looking to contribute an artemis ServiceConnector that would
> > >>>>> leverage
> > >>>>>>> the existing service connector interface to bridge from a queue
> > into a
> > >>>>>>> kafka topic similar to bridge between artemis clusters.
> > >>>>>>>
> > >>>>>>> Idea is to make it configurable how to transform and encode the
> > >>>> message
> > >>>>> in
> > >>>>>>> kafka but as amqp is becoming very well supported by artemis
> > provide
> > >>>> out
> > >>>>>>> the box that the message be encoded in amqp byte protocol.
> > >>>>>>>
> > >>>>>>> It something I’ve been working on privately for a bit, but since
> > >>>>> becoming
> > >>>>>>> a committer here is really like to share it.
> > >>>>>>>
> > >>>>>>> I hope if I get a positive response here to get a branch ready
> > soon,
> > >>>> but
> > >>>>>>> feedback if this is welcomed would be great.
> > >>>>>>>
> > >>>>>>> Cheers
> > >>>>>>> Mike
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Sent from my iPhone
> > >>>>>
> > >>>>
> >
>
Re: Kafka ServiceConnector
Posted by Martyn Taylor <mt...@redhat.com>.
Looking at the Kafka Connect example. I see that the integration
components are all maintained outside of the Kafka project. They're listed
here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
How about we do something similar and create a wiki/web page linking the
integration projects. I realise our website isn't the most accessible atm
but going forward this should be improved.
On Thu, Oct 26, 2017 at 5:42 PM, Robbie Gemmell <ro...@gmail.com>
wrote:
> The related PR is https://github.com/apache/activemq-artemis/pull/1607
> for anyone not following along thus far, or later reference.
>
> My main view there was that such an extension would be better
> maintained seperate from the main broker repository+distribution for a
> variety of reasons. After that I didn't have a strong view on where it
> lives, but I would say that I don't actually see any issue at all with
> it being outwith the ActiveMQ project. To me thats just a key thing
> enabled by such plugin points existing, and not uncommon with other
> projects/components.
>
> Robbie
>
> On 26 October 2017 at 16:46, Michael André Pearce
> <mi...@me.com> wrote:
> > As noted on the PR part of the PR discussion is should Service
> Connectors exist with the broker code.
> >
> > Service Connectors / Extensions are ActiveMQ Artemis specific.
> >
> > I think having them within the project space is good so we can grow an
> eco system of some, people can just use will aid in having better broker
> adoption as tools people look for would be available if contributed.
> >
> > As stated above it seems the general consensus on the PR is though liked
> is the broker code base best for them to live?
> >
> > If we don’t wish them to exist with broker code so they can have their
> own release cycle, and don’t murky the water of the broker, I would like to
> discuss if we could make a sub project for these to live, and possibly
> propose a sub project be setup called “ActiveMQ Artemis Integrations” ?
> >
> > If setup I would also propose maybe moving the spring integration there
> also.
> >
> > Cheers
> > Mike
> >
> >
> >
> >
> >
> >
> >
> > Sent from my iPhone
> >
> >> On 6 Oct 2017, at 12:20, Michael André Pearce <
> michael.andre.pearce@me.com> wrote:
> >>
> >> Yes and no, it would be same as arguing that core bridge between
> artemis clusters could also be separated components.
> >>
> >> The idea is to effectively have similar to a core bridge but to kafka.
> >>
> >> Some key parts why are (and probably similar to why core bridge is
> implemented within the broker)
> >>
> >> Latency
> >> Extra processes and hops
> >> Out the box in broker support
> >>
> >> Cheers
> >> Mike
> >>
> >>
> >> Sent from my iPhone
> >>
> >>> On 6 Oct 2017, at 12:14, Andy Taylor <an...@gmail.com> wrote:
> >>>
> >>> I think https://github.com/ppatierno/kafka-connect-amqp is an example
> of
> >>> this
> >>>
> >>> On 6 October 2017 at 12:07, Christopher Shannon <
> >>> christopher.l.shannon@gmail.com> wrote:
> >>>
> >>>> Why not use Kafka's Connect framework? Seems like building a source
> >>>> connector here would make the most sense. This is not to say I'm
> against
> >>>> the idea I'm just wondering the benefits of doing it from the Artemis
> side
> >>>> of things when Kafka already has a framework that can be used to
> >>>> import/export data.
> >>>>
> >>>> On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce <
> >>>> michael.andre.pearce@me.com> wrote:
> >>>>
> >>>>> Hi Andy,
> >>>>>
> >>>>> As you may be aware or not we run both activemq and kafka each has a
> >>>>> specific plus point to it, use the right tool for the right job.
> >>>>>
> >>>>> We use activemq as our feature rich, fully jms compliant and
> performant
> >>>>> broker.
> >>>>>
> >>>>> And use kafka more for post processing and data streaming, with
> >>>> historical
> >>>>> feed also.
> >>>>>
> >>>>> Idea here is that producers and consumers bind into our activemq
> where
> >>>> jms
> >>>>> api is wanted or needed, lower per message latencies etc. but then
> we can
> >>>>> bridge messages from a core queue into kafka so that the data can
> flow to
> >>>>> it as fast as possible and consumed from there also.
> >>>>>
> >>>>> Idea is to try make a hybrid taking best of both.
> >>>>>
> >>>>> Cheers
> >>>>> Mike
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On 6 Oct 2017, at 09:37, Andy Taylor <an...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> Whats the use case here Michael, sounds like a cool idea and
> something
> >>>>> that
> >>>>>> has been mentioned by different people but I am yet to understand
> how
> >>>>> this
> >>>>>> could be used.
> >>>>>>
> >>>>>> On 6 October 2017 at 07:48, Michael André Pearce <
> >>>>>> michael.andre.pearce@me.com> wrote:
> >>>>>>
> >>>>>>> Hi All,
> >>>>>>>
> >>>>>>> I am looking to contribute an artemis ServiceConnector that would
> >>>>> leverage
> >>>>>>> the existing service connector interface to bridge from a queue
> into a
> >>>>>>> kafka topic similar to bridge between artemis clusters.
> >>>>>>>
> >>>>>>> Idea is to make it configurable how to transform and encode the
> >>>> message
> >>>>> in
> >>>>>>> kafka but as amqp is becoming very well supported by artemis
> provide
> >>>> out
> >>>>>>> the box that the message be encoded in amqp byte protocol.
> >>>>>>>
> >>>>>>> It something I’ve been working on privately for a bit, but since
> >>>>> becoming
> >>>>>>> a committer here is really like to share it.
> >>>>>>>
> >>>>>>> I hope if I get a positive response here to get a branch ready
> soon,
> >>>> but
> >>>>>>> feedback if this is welcomed would be great.
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>> Mike
> >>>>>>>
> >>>>>>>
> >>>>>>> Sent from my iPhone
> >>>>>
> >>>>
>
Re: Kafka ServiceConnector
Posted by Robbie Gemmell <ro...@gmail.com>.
The related PR is https://github.com/apache/activemq-artemis/pull/1607
for anyone not following along thus far, or later reference.
My main view there was that such an extension would be better
maintained seperate from the main broker repository+distribution for a
variety of reasons. After that I didn't have a strong view on where it
lives, but I would say that I don't actually see any issue at all with
it being outwith the ActiveMQ project. To me thats just a key thing
enabled by such plugin points existing, and not uncommon with other
projects/components.
Robbie
On 26 October 2017 at 16:46, Michael André Pearce
<mi...@me.com> wrote:
> As noted on the PR part of the PR discussion is should Service Connectors exist with the broker code.
>
> Service Connectors / Extensions are ActiveMQ Artemis specific.
>
> I think having them within the project space is good so we can grow an eco system of some, people can just use will aid in having better broker adoption as tools people look for would be available if contributed.
>
> As stated above it seems the general consensus on the PR is though liked is the broker code base best for them to live?
>
> If we don’t wish them to exist with broker code so they can have their own release cycle, and don’t murky the water of the broker, I would like to discuss if we could make a sub project for these to live, and possibly propose a sub project be setup called “ActiveMQ Artemis Integrations” ?
>
> If setup I would also propose maybe moving the spring integration there also.
>
> Cheers
> Mike
>
>
>
>
>
>
>
> Sent from my iPhone
>
>> On 6 Oct 2017, at 12:20, Michael André Pearce <mi...@me.com> wrote:
>>
>> Yes and no, it would be same as arguing that core bridge between artemis clusters could also be separated components.
>>
>> The idea is to effectively have similar to a core bridge but to kafka.
>>
>> Some key parts why are (and probably similar to why core bridge is implemented within the broker)
>>
>> Latency
>> Extra processes and hops
>> Out the box in broker support
>>
>> Cheers
>> Mike
>>
>>
>> Sent from my iPhone
>>
>>> On 6 Oct 2017, at 12:14, Andy Taylor <an...@gmail.com> wrote:
>>>
>>> I think https://github.com/ppatierno/kafka-connect-amqp is an example of
>>> this
>>>
>>> On 6 October 2017 at 12:07, Christopher Shannon <
>>> christopher.l.shannon@gmail.com> wrote:
>>>
>>>> Why not use Kafka's Connect framework? Seems like building a source
>>>> connector here would make the most sense. This is not to say I'm against
>>>> the idea I'm just wondering the benefits of doing it from the Artemis side
>>>> of things when Kafka already has a framework that can be used to
>>>> import/export data.
>>>>
>>>> On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce <
>>>> michael.andre.pearce@me.com> wrote:
>>>>
>>>>> Hi Andy,
>>>>>
>>>>> As you may be aware or not we run both activemq and kafka each has a
>>>>> specific plus point to it, use the right tool for the right job.
>>>>>
>>>>> We use activemq as our feature rich, fully jms compliant and performant
>>>>> broker.
>>>>>
>>>>> And use kafka more for post processing and data streaming, with
>>>> historical
>>>>> feed also.
>>>>>
>>>>> Idea here is that producers and consumers bind into our activemq where
>>>> jms
>>>>> api is wanted or needed, lower per message latencies etc. but then we can
>>>>> bridge messages from a core queue into kafka so that the data can flow to
>>>>> it as fast as possible and consumed from there also.
>>>>>
>>>>> Idea is to try make a hybrid taking best of both.
>>>>>
>>>>> Cheers
>>>>> Mike
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On 6 Oct 2017, at 09:37, Andy Taylor <an...@gmail.com> wrote:
>>>>>>
>>>>>> Whats the use case here Michael, sounds like a cool idea and something
>>>>> that
>>>>>> has been mentioned by different people but I am yet to understand how
>>>>> this
>>>>>> could be used.
>>>>>>
>>>>>> On 6 October 2017 at 07:48, Michael André Pearce <
>>>>>> michael.andre.pearce@me.com> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I am looking to contribute an artemis ServiceConnector that would
>>>>> leverage
>>>>>>> the existing service connector interface to bridge from a queue into a
>>>>>>> kafka topic similar to bridge between artemis clusters.
>>>>>>>
>>>>>>> Idea is to make it configurable how to transform and encode the
>>>> message
>>>>> in
>>>>>>> kafka but as amqp is becoming very well supported by artemis provide
>>>> out
>>>>>>> the box that the message be encoded in amqp byte protocol.
>>>>>>>
>>>>>>> It something I’ve been working on privately for a bit, but since
>>>>> becoming
>>>>>>> a committer here is really like to share it.
>>>>>>>
>>>>>>> I hope if I get a positive response here to get a branch ready soon,
>>>> but
>>>>>>> feedback if this is welcomed would be great.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Mike
>>>>>>>
>>>>>>>
>>>>>>> Sent from my iPhone
>>>>>
>>>>
Re: Kafka ServiceConnector
Posted by Michael André Pearce <mi...@me.com>.
As noted on the PR part of the PR discussion is should Service Connectors exist with the broker code.
Service Connectors / Extensions are ActiveMQ Artemis specific.
I think having them within the project space is good so we can grow an eco system of some, people can just use will aid in having better broker adoption as tools people look for would be available if contributed.
As stated above it seems the general consensus on the PR is though liked is the broker code base best for them to live?
If we don’t wish them to exist with broker code so they can have their own release cycle, and don’t murky the water of the broker, I would like to discuss if we could make a sub project for these to live, and possibly propose a sub project be setup called “ActiveMQ Artemis Integrations” ?
If setup I would also propose maybe moving the spring integration there also.
Cheers
Mike
Sent from my iPhone
> On 6 Oct 2017, at 12:20, Michael André Pearce <mi...@me.com> wrote:
>
> Yes and no, it would be same as arguing that core bridge between artemis clusters could also be separated components.
>
> The idea is to effectively have similar to a core bridge but to kafka.
>
> Some key parts why are (and probably similar to why core bridge is implemented within the broker)
>
> Latency
> Extra processes and hops
> Out the box in broker support
>
> Cheers
> Mike
>
>
> Sent from my iPhone
>
>> On 6 Oct 2017, at 12:14, Andy Taylor <an...@gmail.com> wrote:
>>
>> I think https://github.com/ppatierno/kafka-connect-amqp is an example of
>> this
>>
>> On 6 October 2017 at 12:07, Christopher Shannon <
>> christopher.l.shannon@gmail.com> wrote:
>>
>>> Why not use Kafka's Connect framework? Seems like building a source
>>> connector here would make the most sense. This is not to say I'm against
>>> the idea I'm just wondering the benefits of doing it from the Artemis side
>>> of things when Kafka already has a framework that can be used to
>>> import/export data.
>>>
>>> On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce <
>>> michael.andre.pearce@me.com> wrote:
>>>
>>>> Hi Andy,
>>>>
>>>> As you may be aware or not we run both activemq and kafka each has a
>>>> specific plus point to it, use the right tool for the right job.
>>>>
>>>> We use activemq as our feature rich, fully jms compliant and performant
>>>> broker.
>>>>
>>>> And use kafka more for post processing and data streaming, with
>>> historical
>>>> feed also.
>>>>
>>>> Idea here is that producers and consumers bind into our activemq where
>>> jms
>>>> api is wanted or needed, lower per message latencies etc. but then we can
>>>> bridge messages from a core queue into kafka so that the data can flow to
>>>> it as fast as possible and consumed from there also.
>>>>
>>>> Idea is to try make a hybrid taking best of both.
>>>>
>>>> Cheers
>>>> Mike
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 6 Oct 2017, at 09:37, Andy Taylor <an...@gmail.com> wrote:
>>>>>
>>>>> Whats the use case here Michael, sounds like a cool idea and something
>>>> that
>>>>> has been mentioned by different people but I am yet to understand how
>>>> this
>>>>> could be used.
>>>>>
>>>>> On 6 October 2017 at 07:48, Michael André Pearce <
>>>>> michael.andre.pearce@me.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I am looking to contribute an artemis ServiceConnector that would
>>>> leverage
>>>>>> the existing service connector interface to bridge from a queue into a
>>>>>> kafka topic similar to bridge between artemis clusters.
>>>>>>
>>>>>> Idea is to make it configurable how to transform and encode the
>>> message
>>>> in
>>>>>> kafka but as amqp is becoming very well supported by artemis provide
>>> out
>>>>>> the box that the message be encoded in amqp byte protocol.
>>>>>>
>>>>>> It something I’ve been working on privately for a bit, but since
>>>> becoming
>>>>>> a committer here is really like to share it.
>>>>>>
>>>>>> I hope if I get a positive response here to get a branch ready soon,
>>> but
>>>>>> feedback if this is welcomed would be great.
>>>>>>
>>>>>> Cheers
>>>>>> Mike
>>>>>>
>>>>>>
>>>>>> Sent from my iPhone
>>>>
>>>
Re: Kafka ServiceConnector
Posted by Michael André Pearce <mi...@me.com>.
Yes and no, it would be same as arguing that core bridge between artemis clusters could also be separated components.
The idea is to effectively have similar to a core bridge but to kafka.
Some key parts why are (and probably similar to why core bridge is implemented within the broker)
Latency
Extra processes and hops
Out the box in broker support
Cheers
Mike
Sent from my iPhone
> On 6 Oct 2017, at 12:14, Andy Taylor <an...@gmail.com> wrote:
>
> I think https://github.com/ppatierno/kafka-connect-amqp is an example of
> this
>
> On 6 October 2017 at 12:07, Christopher Shannon <
> christopher.l.shannon@gmail.com> wrote:
>
>> Why not use Kafka's Connect framework? Seems like building a source
>> connector here would make the most sense. This is not to say I'm against
>> the idea I'm just wondering the benefits of doing it from the Artemis side
>> of things when Kafka already has a framework that can be used to
>> import/export data.
>>
>> On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce <
>> michael.andre.pearce@me.com> wrote:
>>
>>> Hi Andy,
>>>
>>> As you may be aware or not we run both activemq and kafka each has a
>>> specific plus point to it, use the right tool for the right job.
>>>
>>> We use activemq as our feature rich, fully jms compliant and performant
>>> broker.
>>>
>>> And use kafka more for post processing and data streaming, with
>> historical
>>> feed also.
>>>
>>> Idea here is that producers and consumers bind into our activemq where
>> jms
>>> api is wanted or needed, lower per message latencies etc. but then we can
>>> bridge messages from a core queue into kafka so that the data can flow to
>>> it as fast as possible and consumed from there also.
>>>
>>> Idea is to try make a hybrid taking best of both.
>>>
>>> Cheers
>>> Mike
>>>
>>> Sent from my iPhone
>>>
>>>> On 6 Oct 2017, at 09:37, Andy Taylor <an...@gmail.com> wrote:
>>>>
>>>> Whats the use case here Michael, sounds like a cool idea and something
>>> that
>>>> has been mentioned by different people but I am yet to understand how
>>> this
>>>> could be used.
>>>>
>>>> On 6 October 2017 at 07:48, Michael André Pearce <
>>>> michael.andre.pearce@me.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I am looking to contribute an artemis ServiceConnector that would
>>> leverage
>>>>> the existing service connector interface to bridge from a queue into a
>>>>> kafka topic similar to bridge between artemis clusters.
>>>>>
>>>>> Idea is to make it configurable how to transform and encode the
>> message
>>> in
>>>>> kafka but as amqp is becoming very well supported by artemis provide
>> out
>>>>> the box that the message be encoded in amqp byte protocol.
>>>>>
>>>>> It something I’ve been working on privately for a bit, but since
>>> becoming
>>>>> a committer here is really like to share it.
>>>>>
>>>>> I hope if I get a positive response here to get a branch ready soon,
>> but
>>>>> feedback if this is welcomed would be great.
>>>>>
>>>>> Cheers
>>>>> Mike
>>>>>
>>>>>
>>>>> Sent from my iPhone
>>>
>>
Re: Kafka ServiceConnector
Posted by Andy Taylor <an...@gmail.com>.
I think https://github.com/ppatierno/kafka-connect-amqp is an example of
this
On 6 October 2017 at 12:07, Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> Why not use Kafka's Connect framework? Seems like building a source
> connector here would make the most sense. This is not to say I'm against
> the idea I'm just wondering the benefits of doing it from the Artemis side
> of things when Kafka already has a framework that can be used to
> import/export data.
>
> On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce <
> michael.andre.pearce@me.com> wrote:
>
> > Hi Andy,
> >
> > As you may be aware or not we run both activemq and kafka each has a
> > specific plus point to it, use the right tool for the right job.
> >
> > We use activemq as our feature rich, fully jms compliant and performant
> > broker.
> >
> > And use kafka more for post processing and data streaming, with
> historical
> > feed also.
> >
> > Idea here is that producers and consumers bind into our activemq where
> jms
> > api is wanted or needed, lower per message latencies etc. but then we can
> > bridge messages from a core queue into kafka so that the data can flow to
> > it as fast as possible and consumed from there also.
> >
> > Idea is to try make a hybrid taking best of both.
> >
> > Cheers
> > Mike
> >
> > Sent from my iPhone
> >
> > > On 6 Oct 2017, at 09:37, Andy Taylor <an...@gmail.com> wrote:
> > >
> > > Whats the use case here Michael, sounds like a cool idea and something
> > that
> > > has been mentioned by different people but I am yet to understand how
> > this
> > > could be used.
> > >
> > > On 6 October 2017 at 07:48, Michael André Pearce <
> > > michael.andre.pearce@me.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> I am looking to contribute an artemis ServiceConnector that would
> > leverage
> > >> the existing service connector interface to bridge from a queue into a
> > >> kafka topic similar to bridge between artemis clusters.
> > >>
> > >> Idea is to make it configurable how to transform and encode the
> message
> > in
> > >> kafka but as amqp is becoming very well supported by artemis provide
> out
> > >> the box that the message be encoded in amqp byte protocol.
> > >>
> > >> It something I’ve been working on privately for a bit, but since
> > becoming
> > >> a committer here is really like to share it.
> > >>
> > >> I hope if I get a positive response here to get a branch ready soon,
> but
> > >> feedback if this is welcomed would be great.
> > >>
> > >> Cheers
> > >> Mike
> > >>
> > >>
> > >> Sent from my iPhone
> >
>
Re: Kafka ServiceConnector
Posted by Christopher Shannon <ch...@gmail.com>.
Why not use Kafka's Connect framework? Seems like building a source
connector here would make the most sense. This is not to say I'm against
the idea I'm just wondering the benefits of doing it from the Artemis side
of things when Kafka already has a framework that can be used to
import/export data.
On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce <
michael.andre.pearce@me.com> wrote:
> Hi Andy,
>
> As you may be aware or not we run both activemq and kafka each has a
> specific plus point to it, use the right tool for the right job.
>
> We use activemq as our feature rich, fully jms compliant and performant
> broker.
>
> And use kafka more for post processing and data streaming, with historical
> feed also.
>
> Idea here is that producers and consumers bind into our activemq where jms
> api is wanted or needed, lower per message latencies etc. but then we can
> bridge messages from a core queue into kafka so that the data can flow to
> it as fast as possible and consumed from there also.
>
> Idea is to try make a hybrid taking best of both.
>
> Cheers
> Mike
>
> Sent from my iPhone
>
> > On 6 Oct 2017, at 09:37, Andy Taylor <an...@gmail.com> wrote:
> >
> > Whats the use case here Michael, sounds like a cool idea and something
> that
> > has been mentioned by different people but I am yet to understand how
> this
> > could be used.
> >
> > On 6 October 2017 at 07:48, Michael André Pearce <
> > michael.andre.pearce@me.com> wrote:
> >
> >> Hi All,
> >>
> >> I am looking to contribute an artemis ServiceConnector that would
> leverage
> >> the existing service connector interface to bridge from a queue into a
> >> kafka topic similar to bridge between artemis clusters.
> >>
> >> Idea is to make it configurable how to transform and encode the message
> in
> >> kafka but as amqp is becoming very well supported by artemis provide out
> >> the box that the message be encoded in amqp byte protocol.
> >>
> >> It something I’ve been working on privately for a bit, but since
> becoming
> >> a committer here is really like to share it.
> >>
> >> I hope if I get a positive response here to get a branch ready soon, but
> >> feedback if this is welcomed would be great.
> >>
> >> Cheers
> >> Mike
> >>
> >>
> >> Sent from my iPhone
>
Re: Kafka ServiceConnector
Posted by Michael André Pearce <mi...@me.com>.
Hi Andy,
As you may be aware or not we run both activemq and kafka each has a specific plus point to it, use the right tool for the right job.
We use activemq as our feature rich, fully jms compliant and performant broker.
And use kafka more for post processing and data streaming, with historical feed also.
Idea here is that producers and consumers bind into our activemq where jms api is wanted or needed, lower per message latencies etc. but then we can bridge messages from a core queue into kafka so that the data can flow to it as fast as possible and consumed from there also.
Idea is to try make a hybrid taking best of both.
Cheers
Mike
Sent from my iPhone
> On 6 Oct 2017, at 09:37, Andy Taylor <an...@gmail.com> wrote:
>
> Whats the use case here Michael, sounds like a cool idea and something that
> has been mentioned by different people but I am yet to understand how this
> could be used.
>
> On 6 October 2017 at 07:48, Michael André Pearce <
> michael.andre.pearce@me.com> wrote:
>
>> Hi All,
>>
>> I am looking to contribute an artemis ServiceConnector that would leverage
>> the existing service connector interface to bridge from a queue into a
>> kafka topic similar to bridge between artemis clusters.
>>
>> Idea is to make it configurable how to transform and encode the message in
>> kafka but as amqp is becoming very well supported by artemis provide out
>> the box that the message be encoded in amqp byte protocol.
>>
>> It something I’ve been working on privately for a bit, but since becoming
>> a committer here is really like to share it.
>>
>> I hope if I get a positive response here to get a branch ready soon, but
>> feedback if this is welcomed would be great.
>>
>> Cheers
>> Mike
>>
>>
>> Sent from my iPhone
Re: Kafka ServiceConnector
Posted by Andy Taylor <an...@gmail.com>.
Whats the use case here Michael, sounds like a cool idea and something that
has been mentioned by different people but I am yet to understand how this
could be used.
On 6 October 2017 at 07:48, Michael André Pearce <
michael.andre.pearce@me.com> wrote:
> Hi All,
>
> I am looking to contribute an artemis ServiceConnector that would leverage
> the existing service connector interface to bridge from a queue into a
> kafka topic similar to bridge between artemis clusters.
>
> Idea is to make it configurable how to transform and encode the message in
> kafka but as amqp is becoming very well supported by artemis provide out
> the box that the message be encoded in amqp byte protocol.
>
> It something I’ve been working on privately for a bit, but since becoming
> a committer here is really like to share it.
>
> I hope if I get a positive response here to get a branch ready soon, but
> feedback if this is welcomed would be great.
>
> Cheers
> Mike
>
>
> Sent from my iPhone