[DISCUSSION] Chukwa 0.6 + Hadoop integration

[Eric Yang <ey...@apache.org>]

Hi all,

I like to collect thoughts on plan for branching for Chukwa 0.6 release.
 We have four options:

1. Chukwa 0.6 target for Hadoop 1.x branch.
2. Chukwa 0.6 target for Hadoop 2.x branch.
3. Chukwa 0.6 target to work with either Hadoop 1.x or Hadoop 2.x with
maven profile switching.  Default to Hadoop 1.x branch.
4. Chukwa 0.6 target to work with either Hadoop 1.x or Hadoop 2.x with
maven profile switching.  Default to Hadoop 2.x branch.

Option 2 and 4 may put Chukwa on a longer development path to integrate
with YARN for map reduce job.

My personal preference is option 1 to keep everything simple to help
release often.

regards,
Eric

Re: [DISCUSSION] Chukwa 0.6 + Hadoop integration

[Eric Yang <er...@gmail.com>]

HDFS API has changed a little, and it does not impact Chukwa integration.
 For mapreduce, there are some changes which requires demux job to be
written differently.  This is mainly more code to be written to schedule
map reduce job in YARN framework.  The instrumentation of collecting data
from YARN, will need designs to collect useful data in context of
monitoring YARN.  I agree that 1.x branch seems like a better target for
Chukwa 0.6.

regards,
Eric

On Sun, Dec 1, 2013 at 5:00 PM, Ariel Rabkin <as...@gmail.com> wrote:

> Howdy.
>
> I think option 1 is fine -- let's release what we have for now. My
> impression is that there are still a lot of users on the 1xx branch.
>
> I've been a bit out of the loop on Hadoop development and don't have a
> sense how much pain it is to target different Hadoop versions. Is the
> issue entirely the demux jobs, or did the filesystem APIs also change
> in incompatible ways?
>
> --Ari
>
> On Thu, Nov 28, 2013 at 2:15 PM, Eric Yang <ey...@apache.org> wrote:
> > Hi all,
> >
> > I like to collect thoughts on plan for branching for Chukwa 0.6 release.
> >  We have four options:
> >
> > 1. Chukwa 0.6 target for Hadoop 1.x branch.
> > 2. Chukwa 0.6 target for Hadoop 2.x branch.
> > 3. Chukwa 0.6 target to work with either Hadoop 1.x or Hadoop 2.x with
> > maven profile switching.  Default to Hadoop 1.x branch.
> > 4. Chukwa 0.6 target to work with either Hadoop 1.x or Hadoop 2.x with
> > maven profile switching.  Default to Hadoop 2.x branch.
> >
> > Option 2 and 4 may put Chukwa on a longer development path to integrate
> > with YARN for map reduce job.
> >
> > My personal preference is option 1 to keep everything simple to help
> > release often.
> >
> > regards,
> > Eric
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> Princeton Computer Science Department
>

Re: [DISCUSSION] Chukwa 0.6 + Hadoop integration

[Ariel Rabkin <as...@gmail.com>]

Howdy.

I think option 1 is fine -- let's release what we have for now. My
impression is that there are still a lot of users on the 1xx branch.

I've been a bit out of the loop on Hadoop development and don't have a
sense how much pain it is to target different Hadoop versions. Is the
issue entirely the demux jobs, or did the filesystem APIs also change
in incompatible ways?

--Ari

On Thu, Nov 28, 2013 at 2:15 PM, Eric Yang <ey...@apache.org> wrote:
> Hi all,
>
> I like to collect thoughts on plan for branching for Chukwa 0.6 release.
>  We have four options:
>
> 1. Chukwa 0.6 target for Hadoop 1.x branch.
> 2. Chukwa 0.6 target for Hadoop 2.x branch.
> 3. Chukwa 0.6 target to work with either Hadoop 1.x or Hadoop 2.x with
> maven profile switching.  Default to Hadoop 1.x branch.
> 4. Chukwa 0.6 target to work with either Hadoop 1.x or Hadoop 2.x with
> maven profile switching.  Default to Hadoop 2.x branch.
>
> Option 2 and 4 may put Chukwa on a longer development path to integrate
> with YARN for map reduce job.
>
> My personal preference is option 1 to keep everything simple to help
> release often.
>
> regards,
> Eric



-- 
Ari Rabkin asrabkin@gmail.com
Princeton Computer Science Department