You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Jeremy Custenborder <jc...@gmail.com> on 2012/06/13 22:48:23 UTC
RabbitMQ Source and Sink for Flume-NG
All,
I put together a Flume-NG sink and source for rabbitmq. It has some
error detection support and can backoff, then reestablish a connection
in the event that rabbitmq is unavailable. Feel free to hit me up with
any questions you might have.
The source is released under an apache license.
https://github.com/jcustenborder/flume-ng-rabbitmq
J
Re: RabbitMQ Source and Sink for Flume-NG
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Jeremy,
That is great. Since the source is already under Apache License, you could consider submitting it to the Flume Project itself (unless, there is a dependency on some Apache incompatible license).
Thanks
Hari
--
Hari Shreedharan
On Wednesday, June 13, 2012 at 1:48 PM, Jeremy Custenborder wrote:
> All,
>
> I put together a Flume-NG sink and source for rabbitmq. It has some
> error detection support and can backoff, then reestablish a connection
> in the event that rabbitmq is unavailable. Feel free to hit me up with
> any questions you might have.
>
> The source is released under an apache license.
>
> https://github.com/jcustenborder/flume-ng-rabbitmq
>
> J
Re: RabbitMQ Source and Sink for Flume-NG
Posted by Jeremy Custenborder <jc...@gmail.com>.
Ralph,
Thanks for the links on the license.
This code is specific to RabbitMQ due to the protocol used. It might
work with Apache Qpid which implements AMQP. I just found Apache Qpid
a minute ago when searched for message brokers compatible with AMQP. I
have no clue if it is compatible. That being said the pattern I'm
using could be abstracted out to allow some more code reuse. My only
concern would be the bloat associated with all of the different
libraries needed.
J
On Thu, Jun 14, 2012 at 12:42 AM, Ralph Goers
<ra...@dslextreme.com> wrote:
> There are actually a items that should be considered with this.
> http://www.apache.org/legal/resolved.html#category-a - This itemizes
> licenses that are always OK.
> http://www.apache.org/legal/resolved.html#category-x - This itemizes
> licenses that are not OK - however there are exceptions which I will note in
> a moment.
> http://www.apache.org/legal/resolved.html#category-b - This itemizes
> licenses that are generally OK with some restrictions. The MPL 1.1 falls
> into this bucket.
>
> The exceptions I referenced are that it is generally OK to require build
> tools or similar that are expected to be preinstalled on the system - even
> if they are a category-x license.
> See http://www.apache.org/legal/resolved.html#platform and the FAQ that
> follows. Finally, the FAQ after that, "Can Apache projects rely on
> components whose licensing affects the Apache product?" indicates that
> components that are purely optional can even have dependencies with
> generally prohibited licenses, provided adequate notice is given and they
> are not distributed as part of a release.
>
> Having said that (and I haven't looked at the code), is there a reason the
> sink and source has to be specific to rabbitmq and couldn't apply to Apache
> ActiveMQ as well?
>
> Ralph
>
>
> On Jun 13, 2012, at 5:54 PM, Hari Shreedharan wrote:
>
> Thanks Jeremy. I am not sure of the compatibility. You could take a look at
> this: http://www.apache.org/legal/3party.html. I have not taken a look at
> the code yet. I will look at it sometime later today.
>
> Thanks!
> Hari
>
> --
> Hari Shreedharan
>
> On Wednesday, June 13, 2012 at 5:39 PM, Jeremy Custenborder wrote:
>
> Hari -
> It looks like rabbitmq is licensed via Mozilla Public License v1.1. Is
> that compatible? If it's compatible, I'll create a ticket and submit a
> patch. Are there any structure or code changes you would recommend?
>
> http://www.rabbitmq.com/mpl.html
>
> Senthilvel -
> It would be pretty easy to implement. The only concern I would have is
> the body size of the message. SQS limits the body to 64K
>
>
> On Wed, Jun 13, 2012 at 5:09 PM, Senthilvel Rangaswamy
> <se...@gmail.com> wrote:
>
> This is neat. Now, we need a Amazon SQS sink.
>
>
> On Wed, Jun 13, 2012 at 1:48 PM, Jeremy Custenborder
> <jc...@gmail.com> wrote:
>
>
> All,
>
> I put together a Flume-NG sink and source for rabbitmq. It has some
> error detection support and can backoff, then reestablish a connection
> in the event that rabbitmq is unavailable. Feel free to hit me up with
> any questions you might have.
>
> The source is released under an apache license.
>
> https://github.com/jcustenborder/flume-ng-rabbitmq
>
> J
>
>
>
>
>
> --
> ..Senthil
>
> "If there's anything more important than my ego around, I want it
> caught and shot now."
> - Douglas Adams.
>
>
>
Re: RabbitMQ Source and Sink for Flume-NG
Posted by Ralph Goers <ra...@dslextreme.com>.
There are actually a items that should be considered with this.
http://www.apache.org/legal/resolved.html#category-a - This itemizes licenses that are always OK.
http://www.apache.org/legal/resolved.html#category-x - This itemizes licenses that are not OK - however there are exceptions which I will note in a moment.
http://www.apache.org/legal/resolved.html#category-b - This itemizes licenses that are generally OK with some restrictions. The MPL 1.1 falls into this bucket.
The exceptions I referenced are that it is generally OK to require build tools or similar that are expected to be preinstalled on the system - even if they are a category-x license. See http://www.apache.org/legal/resolved.html#platform and the FAQ that follows. Finally, the FAQ after that, "Can Apache projects rely on components whose licensing affects the Apache product?" indicates that components that are purely optional can even have dependencies with generally prohibited licenses, provided adequate notice is given and they are not distributed as part of a release.
Having said that (and I haven't looked at the code), is there a reason the sink and source has to be specific to rabbitmq and couldn't apply to Apache ActiveMQ as well?
Ralph
On Jun 13, 2012, at 5:54 PM, Hari Shreedharan wrote:
> Thanks Jeremy. I am not sure of the compatibility. You could take a look at this: http://www.apache.org/legal/3party.html. I have not taken a look at the code yet. I will look at it sometime later today.
>
> Thanks!
> Hari
>
> --
> Hari Shreedharan
>
> On Wednesday, June 13, 2012 at 5:39 PM, Jeremy Custenborder wrote:
>
>> Hari -
>> It looks like rabbitmq is licensed via Mozilla Public License v1.1. Is
>> that compatible? If it's compatible, I'll create a ticket and submit a
>> patch. Are there any structure or code changes you would recommend?
>>
>> http://www.rabbitmq.com/mpl.html
>>
>> Senthilvel -
>> It would be pretty easy to implement. The only concern I would have is
>> the body size of the message. SQS limits the body to 64K
>>
>>
>> On Wed, Jun 13, 2012 at 5:09 PM, Senthilvel Rangaswamy
>> <se...@gmail.com> wrote:
>>> This is neat. Now, we need a Amazon SQS sink.
>>>
>>>
>>> On Wed, Jun 13, 2012 at 1:48 PM, Jeremy Custenborder
>>> <jc...@gmail.com> wrote:
>>>>
>>>> All,
>>>>
>>>> I put together a Flume-NG sink and source for rabbitmq. It has some
>>>> error detection support and can backoff, then reestablish a connection
>>>> in the event that rabbitmq is unavailable. Feel free to hit me up with
>>>> any questions you might have.
>>>>
>>>> The source is released under an apache license.
>>>>
>>>> https://github.com/jcustenborder/flume-ng-rabbitmq
>>>>
>>>> J
>>>
>>>
>>>
>>>
>>> --
>>> ..Senthil
>>>
>>> "If there's anything more important than my ego around, I want it
>>> caught and shot now."
>>> - Douglas Adams.
>
Re: RabbitMQ Source and Sink for Flume-NG
Posted by Hari Shreedharan <hs...@cloudera.com>.
Thanks Jeremy. I am not sure of the compatibility. You could take a look at this: http://www.apache.org/legal/3party.html. I have not taken a look at the code yet. I will look at it sometime later today.
Thanks!
Hari
--
Hari Shreedharan
On Wednesday, June 13, 2012 at 5:39 PM, Jeremy Custenborder wrote:
> Hari -
> It looks like rabbitmq is licensed via Mozilla Public License v1.1. Is
> that compatible? If it's compatible, I'll create a ticket and submit a
> patch. Are there any structure or code changes you would recommend?
>
> http://www.rabbitmq.com/mpl.html
>
> Senthilvel -
> It would be pretty easy to implement. The only concern I would have is
> the body size of the message. SQS limits the body to 64K
>
>
> On Wed, Jun 13, 2012 at 5:09 PM, Senthilvel Rangaswamy
> <senthilvel@gmail.com (mailto:senthilvel@gmail.com)> wrote:
> > This is neat. Now, we need a Amazon SQS sink.
> >
> >
> > On Wed, Jun 13, 2012 at 1:48 PM, Jeremy Custenborder
> > <jcustenborder@gmail.com (mailto:jcustenborder@gmail.com)> wrote:
> > >
> > > All,
> > >
> > > I put together a Flume-NG sink and source for rabbitmq. It has some
> > > error detection support and can backoff, then reestablish a connection
> > > in the event that rabbitmq is unavailable. Feel free to hit me up with
> > > any questions you might have.
> > >
> > > The source is released under an apache license.
> > >
> > > https://github.com/jcustenborder/flume-ng-rabbitmq
> > >
> > > J
> >
> >
> >
> >
> > --
> > ..Senthil
> >
> > "If there's anything more important than my ego around, I want it
> > caught and shot now."
> > - Douglas Adams.
> >
>
>
>
Re: RabbitMQ Source and Sink for Flume-NG
Posted by Jeremy Custenborder <jc...@gmail.com>.
Hari -
It looks like rabbitmq is licensed via Mozilla Public License v1.1. Is
that compatible? If it's compatible, I'll create a ticket and submit a
patch. Are there any structure or code changes you would recommend?
http://www.rabbitmq.com/mpl.html
Senthilvel -
It would be pretty easy to implement. The only concern I would have is
the body size of the message. SQS limits the body to 64K
On Wed, Jun 13, 2012 at 5:09 PM, Senthilvel Rangaswamy
<se...@gmail.com> wrote:
> This is neat. Now, we need a Amazon SQS sink.
>
>
> On Wed, Jun 13, 2012 at 1:48 PM, Jeremy Custenborder
> <jc...@gmail.com> wrote:
>>
>> All,
>>
>> I put together a Flume-NG sink and source for rabbitmq. It has some
>> error detection support and can backoff, then reestablish a connection
>> in the event that rabbitmq is unavailable. Feel free to hit me up with
>> any questions you might have.
>>
>> The source is released under an apache license.
>>
>> https://github.com/jcustenborder/flume-ng-rabbitmq
>>
>> J
>
>
>
>
> --
> ..Senthil
>
> "If there's anything more important than my ego around, I want it
> caught and shot now."
> - Douglas Adams.
>
Re: RabbitMQ Source and Sink for Flume-NG
Posted by Senthilvel Rangaswamy <se...@gmail.com>.
This is neat. Now, we need a Amazon SQS sink.
On Wed, Jun 13, 2012 at 1:48 PM, Jeremy Custenborder <
jcustenborder@gmail.com> wrote:
> All,
>
> I put together a Flume-NG sink and source for rabbitmq. It has some
> error detection support and can backoff, then reestablish a connection
> in the event that rabbitmq is unavailable. Feel free to hit me up with
> any questions you might have.
>
> The source is released under an apache license.
>
> https://github.com/jcustenborder/flume-ng-rabbitmq
>
> J
>
--
..Senthil
"If there's anything more important than my ego around, I want it
caught and shot now."
- Douglas Adams.