customizable avro source/sink events - flume-918

[Karthik K <os...@gmail.com>]

Team -
   For a particular use case scenario, there is a need to subscribe to a
custom avro stream and route it to flume , to be consumed by other
downstream sink(s). The avro (*.avdl) published by the stream is different
from the src/main/avro/flume.avdl , reference implementation as part of the
distribution .  Would like to reuse most of the reference implementation
present in the form of AvroSource / AvroSink. except for the
marshalling/unmarshalling that is custom to our needs.

   Towards that, flume-918 has been logged and  a patch is up, that
separates the marshalling / unmarshalling of flume events separate from the
avro reference implementation present in the same.

   There must be a corresponding reviewboard ticket logged for the same,
and Eric had put in his comments and a revised patch should be up in
reviewboard as well, post the same.

   Given the nature of the same, the patch can certainly use more reviews ,
as the possible usage of the same can extend beyond just the
avrosource/sink that is being dealt with.   Thanks !

--
  Karthik.

Re: customizable avro source/sink events - flume-918

[Karthik K <os...@gmail.com>]

A revised patch compatible with the recent updates is now in, that helps
the source/sink developer to focus on the event marshalling /unmarshalling
while leaving the binding/communicating semantics abstracted away as part
of flume, to help with users reuse a good chunk of it.


Would appreciate design comments / reviews on the same, given the nature of
it . Thanks for your time !!


On Fri, Jan 13, 2012 at 4:58 PM, Karthik K <os...@gmail.com> wrote:

> Team -
>    For a particular use case scenario, there is a need to subscribe to a
> custom avro stream and route it to flume , to be consumed by other
> downstream sink(s). The avro (*.avdl) published by the stream is different
> from the src/main/avro/flume.avdl , reference implementation as part of the
> distribution .  Would like to reuse most of the reference implementation
> present in the form of AvroSource / AvroSink. except for the
> marshalling/unmarshalling that is custom to our needs.
>
>    Towards that, flume-918 has been logged and  a patch is up, that
> separates the marshalling / unmarshalling of flume events separate from the
> avro reference implementation present in the same.
>
>    There must be a corresponding reviewboard ticket logged for the same,
> and Eric had put in his comments and a revised patch should be up in
> reviewboard as well, post the same.
>
>    Given the nature of the same, the patch can certainly use more reviews
> , as the possible usage of the same can extend beyond just the
> avrosource/sink that is being dealt with.   Thanks !
>
> --
>   Karthik.
>
>