You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Ralph Goers <ra...@dslextreme.com> on 2011/12/31 09:04:08 UTC
Events in NG
Was it really necessary to make the Event object in NG backward incompatible. This will make upgrading systems very painful as everything needs to be upgraded at once.
Ralph
Re: Events in NG
Posted by Eric Sammer <es...@cloudera.com>.
In support of this, I've started a page on the wiki to track a
compatibility proposal.
http://bit.ly/tDByZf
I've also setup a section to track Flume Enhancement Proposals (similar to
the way Avro and Python track things like this) for these kinds of things
in the future.
http://bit.ly/rsxKiB
On Mon, Jan 2, 2012 at 2:09 PM, Eric Sammer <es...@cloudera.com> wrote:
> On Sat, Dec 31, 2011 at 7:38 AM, Ralph Goers <ra...@dslextreme.com>wrote:
>
>> I guess I don't see completely how that can work. In OG the event
>> contained getTimestamp(), getNanos() and getHost(). The Event in OG does
>> not have these fields.
>
>
> These accessors were just sugar for poking things into the headers of the
> event. The equivalent would be to explicitly stash these attributes in the
> headers of the event so they're available down the line (i.e. in the sink
> to which they're sent).
>
>
>> We have an application that is reading legacy log data, creating an OG
>> event and passing it through Flume to a custom Sink to write to Cassandra.
>
>
> Sources and sinks obviously need to be updated to the new APIs at which
> time they should use headers directly.
>
>
>> In addition, we will have a bunch of apps that use an Appender to create
>> an OG Event which will pass it to an agent to forward to the same Cassandra
>> sink. I'm trying to figure out how we can do the upgrade to NG in a sane
>> fashion.
>>
>
> The log4j / logback appender case is a bit trickier. Basically, you do
> need to update the appender and the agent to which it writes at the same
> time or have it speak to the proposed compatibility source in an NG agent
> which knows how to deal with OG thrift events.
>
> I plan on writing up a compatibility proposal on the wiki as a strawman.
> If that makes sense, we'll render it out in JIRAs and run with it. If it
> doesn't, we'll work together to figure out a better plan. I was hoping to
> get away without spending a lot of time on compatibility with OG, but if
> the community as a whole feels phased upgrades are going to be critical I'm
> happy to help work out how it might go.
>
>
>> Ralph
>>
>> On Dec 31, 2011, at 1:30 AM, Eric Sammer wrote:
>>
>> > I've been running through some of the compatibility scenarios and while
>> I
>> > haven't spent a ton of time on it, I think there's a solution. It's true
>> > that the event object isn't the same as OG. That would have been bulky
>> as
>> > we would have had to include thrift just to accommodate it (NG's event
>> is
>> > just a pojo to prevent RPC bleed throughout the codebase). Since users
>> can
>> > only upgrade an entire agent at once - that is, run an OG agent and
>> connect
>> > it to an NG agent or vice versa - we could introduce a compatibility
>> source
>> > and sink for NG and handle the marshaling from OG events to NG events.
>> >
>> > I don't know if that fully allays your concern, but it should get us to
>> > where we need to be.
>> >
>> > On Sat, Dec 31, 2011 at 12:04 AM, Ralph Goers <
>> ralph.goers@dslextreme.com>wrote:
>> >
>> >> Was it really necessary to make the Event object in NG backward
>> >> incompatible. This will make upgrading systems very painful as
>> everything
>> >> needs to be upgraded at once.
>> >>
>> >> Ralph
>> >
>> >
>> >
>> >
>> > --
>> > Eric Sammer
>> > twitter: esammer
>> > data: www.cloudera.com
>>
>>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
Re: Events in NG
Posted by Eric Sammer <es...@cloudera.com>.
On Sat, Dec 31, 2011 at 7:38 AM, Ralph Goers <ra...@dslextreme.com>wrote:
> I guess I don't see completely how that can work. In OG the event
> contained getTimestamp(), getNanos() and getHost(). The Event in OG does
> not have these fields.
These accessors were just sugar for poking things into the headers of the
event. The equivalent would be to explicitly stash these attributes in the
headers of the event so they're available down the line (i.e. in the sink
to which they're sent).
> We have an application that is reading legacy log data, creating an OG
> event and passing it through Flume to a custom Sink to write to Cassandra.
Sources and sinks obviously need to be updated to the new APIs at which
time they should use headers directly.
> In addition, we will have a bunch of apps that use an Appender to create
> an OG Event which will pass it to an agent to forward to the same Cassandra
> sink. I'm trying to figure out how we can do the upgrade to NG in a sane
> fashion.
>
The log4j / logback appender case is a bit trickier. Basically, you do need
to update the appender and the agent to which it writes at the same time or
have it speak to the proposed compatibility source in an NG agent which
knows how to deal with OG thrift events.
I plan on writing up a compatibility proposal on the wiki as a strawman. If
that makes sense, we'll render it out in JIRAs and run with it. If it
doesn't, we'll work together to figure out a better plan. I was hoping to
get away without spending a lot of time on compatibility with OG, but if
the community as a whole feels phased upgrades are going to be critical I'm
happy to help work out how it might go.
> Ralph
>
> On Dec 31, 2011, at 1:30 AM, Eric Sammer wrote:
>
> > I've been running through some of the compatibility scenarios and while I
> > haven't spent a ton of time on it, I think there's a solution. It's true
> > that the event object isn't the same as OG. That would have been bulky as
> > we would have had to include thrift just to accommodate it (NG's event is
> > just a pojo to prevent RPC bleed throughout the codebase). Since users
> can
> > only upgrade an entire agent at once - that is, run an OG agent and
> connect
> > it to an NG agent or vice versa - we could introduce a compatibility
> source
> > and sink for NG and handle the marshaling from OG events to NG events.
> >
> > I don't know if that fully allays your concern, but it should get us to
> > where we need to be.
> >
> > On Sat, Dec 31, 2011 at 12:04 AM, Ralph Goers <
> ralph.goers@dslextreme.com>wrote:
> >
> >> Was it really necessary to make the Event object in NG backward
> >> incompatible. This will make upgrading systems very painful as
> everything
> >> needs to be upgraded at once.
> >>
> >> Ralph
> >
> >
> >
> >
> > --
> > Eric Sammer
> > twitter: esammer
> > data: www.cloudera.com
>
>
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
Re: Events in NG
Posted by Ralph Goers <ra...@dslextreme.com>.
I guess I don't see completely how that can work. In OG the event contained getTimestamp(), getNanos() and getHost(). The Event in OG does not have these fields. We have an application that is reading legacy log data, creating an OG event and passing it through Flume to a custom Sink to write to Cassandra. In addition, we will have a bunch of apps that use an Appender to create an OG Event which will pass it to an agent to forward to the same Cassandra sink. I'm trying to figure out how we can do the upgrade to NG in a sane fashion.
Ralph
On Dec 31, 2011, at 1:30 AM, Eric Sammer wrote:
> I've been running through some of the compatibility scenarios and while I
> haven't spent a ton of time on it, I think there's a solution. It's true
> that the event object isn't the same as OG. That would have been bulky as
> we would have had to include thrift just to accommodate it (NG's event is
> just a pojo to prevent RPC bleed throughout the codebase). Since users can
> only upgrade an entire agent at once - that is, run an OG agent and connect
> it to an NG agent or vice versa - we could introduce a compatibility source
> and sink for NG and handle the marshaling from OG events to NG events.
>
> I don't know if that fully allays your concern, but it should get us to
> where we need to be.
>
> On Sat, Dec 31, 2011 at 12:04 AM, Ralph Goers <ra...@dslextreme.com>wrote:
>
>> Was it really necessary to make the Event object in NG backward
>> incompatible. This will make upgrading systems very painful as everything
>> needs to be upgraded at once.
>>
>> Ralph
>
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
Re: Events in NG
Posted by Eric Sammer <es...@cloudera.com>.
I've been running through some of the compatibility scenarios and while I
haven't spent a ton of time on it, I think there's a solution. It's true
that the event object isn't the same as OG. That would have been bulky as
we would have had to include thrift just to accommodate it (NG's event is
just a pojo to prevent RPC bleed throughout the codebase). Since users can
only upgrade an entire agent at once - that is, run an OG agent and connect
it to an NG agent or vice versa - we could introduce a compatibility source
and sink for NG and handle the marshaling from OG events to NG events.
I don't know if that fully allays your concern, but it should get us to
where we need to be.
On Sat, Dec 31, 2011 at 12:04 AM, Ralph Goers <ra...@dslextreme.com>wrote:
> Was it really necessary to make the Event object in NG backward
> incompatible. This will make upgrading systems very painful as everything
> needs to be upgraded at once.
>
> Ralph
--
Eric Sammer
twitter: esammer
data: www.cloudera.com