[Discussion] Move to maven to make chukwa more modular

[Eric Yang <ey...@yahoo-inc.com>]

Hi,

For making chukwa more user friendly, it may be worthwhile to convert the project into multiple sub modules.  Hence, interested party could pick and choose which module to build and deploy.  Maven seems like a good choice to manage dependencies among submodules and also improve integration with hadoop and hbase versions.

The layout could look like this:

Trunk
  /common
  /agent
  /collector
  /demux
  /analytics
  /hicc

Demux is the combination of data processors.  It could be named "extractions" as well. Collector module depends on demux module for HBaseWriter.

Since there is no official release of hadoop + hbase that work together.  It will take some time for both projects to stabilize.  I think we can make this change while they are evolving.  Chukwa 0.5 release can come in the time when both projects released official versions then we release 0.5.  Thoughts?

Regards,
Eric

Re: [Discussion] Move to maven to make chukwa more modular

[Eric Yang <ey...@yahoo-inc.com>]

Advantage of maven, we will be able to publish our jar file into maven repository for other projects to adopt our software.  It also cuts a cleaner separation of module dependencies for packaging.  In the case of separating config, maven can express the packaging once, and not repeating in super pom.  In ant, this needs to be setup manually, and build target dependency may not seem clear over time.

Regards,
Eric

On 3/21/11 3:07 PM, "Bill Graham" <bi...@gmail.com> wrote:

+1 on being able to build specific modules in Chukwa. I think it would
be great to be able to build each separately, along with only the
config files that each needs (we went through some pains to do this).

It seems like we could just use ant for this without having to make
major changes to the directory layout. Maybe we split the configs up
into sub-dirs, but do we need much more? It seems like the source
packages are already fairly well delineated.

If we don't want to wait on Hadoop/HBase, we could always release a
developer build of 4.x.



On Mon, Mar 21, 2011 at 2:10 PM, Ariel Rabkin <as...@gmail.com> wrote:
> I think maven would be good but isn't essential.
> My inclination is to take our current trunk and try releasing it, just
> to get familiar with the incubator release process and to help build
> momentum.
>
> --Ari
>
> On Mon, Mar 21, 2011 at 10:43 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
>> Hi,
>>
>> For making chukwa more user friendly, it may be worthwhile to convert the project into multiple sub modules.  Hence, interested party could pick and choose which module to build and deploy.  Maven seems like a good choice to manage dependencies among submodules and also improve integration with hadoop and hbase versions.
>>
>> The layout could look like this:
>>
>> Trunk
>>  /common
>>  /agent
>>  /collector
>>  /demux
>>  /analytics
>>  /hicc
>>
>> Demux is the combination of data processors.  It could be named "extractions" as well. Collector module depends on demux module for HBaseWriter.
>>
>> Since there is no official release of hadoop + hbase that work together.  It will take some time for both projects to stabilize.  I think we can make this change while they are evolving.  Chukwa 0.5 release can come in the time when both projects released official versions then we release 0.5.  Thoughts?
>>
>> Regards,
>> Eric
>>
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>

Re: [Discussion] Move to maven to make chukwa more modular

[Bill Graham <bi...@gmail.com>]

+1 on being able to build specific modules in Chukwa. I think it would
be great to be able to build each separately, along with only the
config files that each needs (we went through some pains to do this).

It seems like we could just use ant for this without having to make
major changes to the directory layout. Maybe we split the configs up
into sub-dirs, but do we need much more? It seems like the source
packages are already fairly well delineated.

If we don't want to wait on Hadoop/HBase, we could always release a
developer build of 4.x.



On Mon, Mar 21, 2011 at 2:10 PM, Ariel Rabkin <as...@gmail.com> wrote:
> I think maven would be good but isn't essential.
> My inclination is to take our current trunk and try releasing it, just
> to get familiar with the incubator release process and to help build
> momentum.
>
> --Ari
>
> On Mon, Mar 21, 2011 at 10:43 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
>> Hi,
>>
>> For making chukwa more user friendly, it may be worthwhile to convert the project into multiple sub modules.  Hence, interested party could pick and choose which module to build and deploy.  Maven seems like a good choice to manage dependencies among submodules and also improve integration with hadoop and hbase versions.
>>
>> The layout could look like this:
>>
>> Trunk
>>  /common
>>  /agent
>>  /collector
>>  /demux
>>  /analytics
>>  /hicc
>>
>> Demux is the combination of data processors.  It could be named "extractions" as well. Collector module depends on demux module for HBaseWriter.
>>
>> Since there is no official release of hadoop + hbase that work together.  It will take some time for both projects to stabilize.  I think we can make this change while they are evolving.  Chukwa 0.5 release can come in the time when both projects released official versions then we release 0.5.  Thoughts?
>>
>> Regards,
>> Eric
>>
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>

Re: [Discussion] Move to maven to make chukwa more modular

[Ariel Rabkin <as...@gmail.com>]

I think maven would be good but isn't essential.
My inclination is to take our current trunk and try releasing it, just
to get familiar with the incubator release process and to help build
momentum.

--Ari

On Mon, Mar 21, 2011 at 10:43 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
> Hi,
>
> For making chukwa more user friendly, it may be worthwhile to convert the project into multiple sub modules.  Hence, interested party could pick and choose which module to build and deploy.  Maven seems like a good choice to manage dependencies among submodules and also improve integration with hadoop and hbase versions.
>
> The layout could look like this:
>
> Trunk
>  /common
>  /agent
>  /collector
>  /demux
>  /analytics
>  /hicc
>
> Demux is the combination of data processors.  It could be named "extractions" as well. Collector module depends on demux module for HBaseWriter.
>
> Since there is no official release of hadoop + hbase that work together.  It will take some time for both projects to stabilize.  I think we can make this change while they are evolving.  Chukwa 0.5 release can come in the time when both projects released official versions then we release 0.5.  Thoughts?
>
> Regards,
> Eric
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department