You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Joey Echeverria <jo...@cloudera.com> on 2013/07/26 17:02:51 UTC
Hadoop 2.0 Support for Accumulo 1.4 Branch
Cloudera announced last night our support for Accumulo 1.4.3 on CDH4:
http://www.slideshare.net/JoeyEcheverria/apache-accumulo-and-cloudera
This required back porting about 11 patches in whole or in part from the
1.5 line on top of 1.4.3. Our release is still in a semi-private beta, but
when it's fully public it will be downloadable along with all of the extra
patches that we committed.
My question is if the community would be interested in us pulling those
back ports upstream?
I believe this would violate the previously agreed upon rule of no feature
back ports to 1.4.3, depending on how we "label" support for Hadoop 2.0.
Thoughts?
-Joey
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Josh Elser <jo...@gmail.com>.
+1
On 7/26/13 11:25 AM, Eric Newton wrote:
> "My question is if the community would be interested in us pulling those
> back ports upstream?"
>
> Yes, please.
>
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Eric Newton <er...@gmail.com>.
"My question is if the community would be interested in us pulling those
back ports upstream?"
Yes, please.
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Billie Rinaldi <bi...@gmail.com>.
On Aug 2, 2013 2:22 PM, "Joey Echeverria" <jo...@cloudera.com> wrote:
>
> Sorry for the delay, it's been one of those weeks.
>
> The current version would probably not be backwards compatible to
> 0.20.2 just based on changes in dependencies. We're looking right now
> to see how hard it is to have three way compatibility (0.20, 1.0,
> 2.0).
It may not help much, but I believe the minimum dependency for 1.4 was
bumped up to 0.20.205.0.
Billie
>
> -Joey
>
> On Thu, Aug 1, 2013 at 7:33 PM, Dave Marion <dl...@comcast.net> wrote:
> > Any update?
> >
> > -----Original Message-----
> > From: Joey Echeverria [mailto:joey@cloudera.com]
> > Sent: Monday, July 29, 2013 1:24 PM
> > To: dev@accumulo.apache.org
> > Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
> >
> > We're testing this today. I'll report back what we find.
> >
> >
> > -Joey
> > —
> > Sent from Mailbox for iPhone
> >
> > On Fri, Jul 26, 2013 at 3:34 PM, null <dl...@comcast.net> wrote:
> >
> >> "Will 1.4 still work with 0.20 with these patches?"
> >> Great point Billie.
> >> ----- Original Message -----
> >> From: "Billie Rinaldi" <bi...@gmail.com>
> >> To: dev@accumulo.apache.org
> >> Sent: Friday, July 26, 2013 3:02:41 PM
> >> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch On Fri, Jul
> >> 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
> >>> > If these patches are going to be included with 1.4.4 or 1.4.5, I
> >>> > would
> >>> like
> >>> > to see the following test run using CDH4 on at least a 5 node
cluster.
> >>> > More nodes would be better.
> >>> >
> >>> > * unit test
> >>> > * Functional test
> >>> > * 24 hr Continuous ingest + verification
> >>> > * 24 hr Continuous ingest + verification + agitation
> >>> > * 24 hr Random walk
> >>> > * 24 hr Random walk + agitation
> >>> >
> >>> > I may be able to assist with this, but I can not make any promises.
> >>>
> >>> Sure thing. Is there already a write-up on running this full battery
> >>> of tests? I have a 10 node cluster that I can use for this.
> >>>
> >>>
> >>> > Great. I think this would be a good patch for 1.4. I assume that
> >>> > if a user stays with Hadoop 1 there are no dependency changes?
> >>>
> >>> Yup. It works the same way as 1.5 where all of the dependency changes
> >>> are in a Hadoop 2.0 profile.
> >>>
> >> In 1.5.0, we gave up on compatibility with 0.20 (and early versions of
> >> 1.0) to make the compatibility requirements simpler; we ended up
> >> without dependency changes in the hadoop version profiles. Will 1.4
> >> still work with 0.20 with these patches? If there are dependency
> >> changes in the profiles, 1.4 would have to be compiled against a
> >> hadoop version compatible with the running version of hadoop, correct?
> >> We had some trouble in the
> >> 1.5 release process with figuring out how to provide multiple binary
> >> artifacts (each compiled against a different version of hadoop) for
> >> the same release. Just something we should consider before we are in
> >> the midst of releasing 1.4.4.
> >> Billie
> >>> -Joey
> >>>
> >
>
>
>
> --
> Joey Echeverria
> Director, Federal FTS
> Cloudera, Inc.
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Joey Echeverria <jo...@cloudera.com>.
Vanilla.
On Fri, Aug 2, 2013 at 3:58 PM, Mike Drob <md...@mdrob.com> wrote:
> Which version of 0.20 are you testing against? Vanilla, or cdh3 flavored?
>
>
> On Fri, Aug 2, 2013 at 2:37 PM, Joey Echeverria <jo...@cloudera.com> wrote:
>
>> I don't think that's a good idea unless you can come up with very
>> clear version number change.
>>
>> -Joey
>>
>> On Fri, Aug 2, 2013 at 2:31 PM, Christopher <ct...@apache.org> wrote:
>> > Would it be reasonable to consider a version of 1.4 that breaks
>> > compatibility with 0.20? I'm not really a fan of this, personally, but
>> > am curious what others think.
>> >
>> > --
>> > Christopher L Tubbs II
>> > http://gravatar.com/ctubbsii
>> >
>> >
>> > On Fri, Aug 2, 2013 at 2:22 PM, Joey Echeverria <jo...@cloudera.com>
>> wrote:
>> >> Sorry for the delay, it's been one of those weeks.
>> >>
>> >> The current version would probably not be backwards compatible to
>> >> 0.20.2 just based on changes in dependencies. We're looking right now
>> >> to see how hard it is to have three way compatibility (0.20, 1.0,
>> >> 2.0).
>> >>
>> >> -Joey
>> >>
>> >> On Thu, Aug 1, 2013 at 7:33 PM, Dave Marion <dl...@comcast.net>
>> wrote:
>> >>> Any update?
>> >>>
>> >>> -----Original Message-----
>> >>> From: Joey Echeverria [mailto:joey@cloudera.com]
>> >>> Sent: Monday, July 29, 2013 1:24 PM
>> >>> To: dev@accumulo.apache.org
>> >>> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
>> >>>
>> >>> We're testing this today. I'll report back what we find.
>> >>>
>> >>>
>> >>> -Joey
>> >>> —
>> >>> Sent from Mailbox for iPhone
>> >>>
>> >>> On Fri, Jul 26, 2013 at 3:34 PM, null <dl...@comcast.net> wrote:
>> >>>
>> >>>> "Will 1.4 still work with 0.20 with these patches?"
>> >>>> Great point Billie.
>> >>>> ----- Original Message -----
>> >>>> From: "Billie Rinaldi" <bi...@gmail.com>
>> >>>> To: dev@accumulo.apache.org
>> >>>> Sent: Friday, July 26, 2013 3:02:41 PM
>> >>>> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch On Fri, Jul
>> >>>> 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
>> >>>>> > If these patches are going to be included with 1.4.4 or 1.4.5, I
>> >>>>> > would
>> >>>>> like
>> >>>>> > to see the following test run using CDH4 on at least a 5 node
>> cluster.
>> >>>>> > More nodes would be better.
>> >>>>> >
>> >>>>> > * unit test
>> >>>>> > * Functional test
>> >>>>> > * 24 hr Continuous ingest + verification
>> >>>>> > * 24 hr Continuous ingest + verification + agitation
>> >>>>> > * 24 hr Random walk
>> >>>>> > * 24 hr Random walk + agitation
>> >>>>> >
>> >>>>> > I may be able to assist with this, but I can not make any promises.
>> >>>>>
>> >>>>> Sure thing. Is there already a write-up on running this full battery
>> >>>>> of tests? I have a 10 node cluster that I can use for this.
>> >>>>>
>> >>>>>
>> >>>>> > Great. I think this would be a good patch for 1.4. I assume that
>> >>>>> > if a user stays with Hadoop 1 there are no dependency changes?
>> >>>>>
>> >>>>> Yup. It works the same way as 1.5 where all of the dependency changes
>> >>>>> are in a Hadoop 2.0 profile.
>> >>>>>
>> >>>> In 1.5.0, we gave up on compatibility with 0.20 (and early versions of
>> >>>> 1.0) to make the compatibility requirements simpler; we ended up
>> >>>> without dependency changes in the hadoop version profiles. Will 1.4
>> >>>> still work with 0.20 with these patches? If there are dependency
>> >>>> changes in the profiles, 1.4 would have to be compiled against a
>> >>>> hadoop version compatible with the running version of hadoop, correct?
>> >>>> We had some trouble in the
>> >>>> 1.5 release process with figuring out how to provide multiple binary
>> >>>> artifacts (each compiled against a different version of hadoop) for
>> >>>> the same release. Just something we should consider before we are in
>> >>>> the midst of releasing 1.4.4.
>> >>>> Billie
>> >>>>> -Joey
>> >>>>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Joey Echeverria
>> >> Director, Federal FTS
>> >> Cloudera, Inc.
>>
>>
>>
>> --
>> Joey Echeverria
>> Director, Federal FTS
>> Cloudera, Inc.
>>
--
Joey Echeverria
Director, Federal FTS
Cloudera, Inc.
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Mike Drob <md...@mdrob.com>.
Which version of 0.20 are you testing against? Vanilla, or cdh3 flavored?
On Fri, Aug 2, 2013 at 2:37 PM, Joey Echeverria <jo...@cloudera.com> wrote:
> I don't think that's a good idea unless you can come up with very
> clear version number change.
>
> -Joey
>
> On Fri, Aug 2, 2013 at 2:31 PM, Christopher <ct...@apache.org> wrote:
> > Would it be reasonable to consider a version of 1.4 that breaks
> > compatibility with 0.20? I'm not really a fan of this, personally, but
> > am curious what others think.
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> >
> > On Fri, Aug 2, 2013 at 2:22 PM, Joey Echeverria <jo...@cloudera.com>
> wrote:
> >> Sorry for the delay, it's been one of those weeks.
> >>
> >> The current version would probably not be backwards compatible to
> >> 0.20.2 just based on changes in dependencies. We're looking right now
> >> to see how hard it is to have three way compatibility (0.20, 1.0,
> >> 2.0).
> >>
> >> -Joey
> >>
> >> On Thu, Aug 1, 2013 at 7:33 PM, Dave Marion <dl...@comcast.net>
> wrote:
> >>> Any update?
> >>>
> >>> -----Original Message-----
> >>> From: Joey Echeverria [mailto:joey@cloudera.com]
> >>> Sent: Monday, July 29, 2013 1:24 PM
> >>> To: dev@accumulo.apache.org
> >>> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
> >>>
> >>> We're testing this today. I'll report back what we find.
> >>>
> >>>
> >>> -Joey
> >>> —
> >>> Sent from Mailbox for iPhone
> >>>
> >>> On Fri, Jul 26, 2013 at 3:34 PM, null <dl...@comcast.net> wrote:
> >>>
> >>>> "Will 1.4 still work with 0.20 with these patches?"
> >>>> Great point Billie.
> >>>> ----- Original Message -----
> >>>> From: "Billie Rinaldi" <bi...@gmail.com>
> >>>> To: dev@accumulo.apache.org
> >>>> Sent: Friday, July 26, 2013 3:02:41 PM
> >>>> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch On Fri, Jul
> >>>> 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
> >>>>> > If these patches are going to be included with 1.4.4 or 1.4.5, I
> >>>>> > would
> >>>>> like
> >>>>> > to see the following test run using CDH4 on at least a 5 node
> cluster.
> >>>>> > More nodes would be better.
> >>>>> >
> >>>>> > * unit test
> >>>>> > * Functional test
> >>>>> > * 24 hr Continuous ingest + verification
> >>>>> > * 24 hr Continuous ingest + verification + agitation
> >>>>> > * 24 hr Random walk
> >>>>> > * 24 hr Random walk + agitation
> >>>>> >
> >>>>> > I may be able to assist with this, but I can not make any promises.
> >>>>>
> >>>>> Sure thing. Is there already a write-up on running this full battery
> >>>>> of tests? I have a 10 node cluster that I can use for this.
> >>>>>
> >>>>>
> >>>>> > Great. I think this would be a good patch for 1.4. I assume that
> >>>>> > if a user stays with Hadoop 1 there are no dependency changes?
> >>>>>
> >>>>> Yup. It works the same way as 1.5 where all of the dependency changes
> >>>>> are in a Hadoop 2.0 profile.
> >>>>>
> >>>> In 1.5.0, we gave up on compatibility with 0.20 (and early versions of
> >>>> 1.0) to make the compatibility requirements simpler; we ended up
> >>>> without dependency changes in the hadoop version profiles. Will 1.4
> >>>> still work with 0.20 with these patches? If there are dependency
> >>>> changes in the profiles, 1.4 would have to be compiled against a
> >>>> hadoop version compatible with the running version of hadoop, correct?
> >>>> We had some trouble in the
> >>>> 1.5 release process with figuring out how to provide multiple binary
> >>>> artifacts (each compiled against a different version of hadoop) for
> >>>> the same release. Just something we should consider before we are in
> >>>> the midst of releasing 1.4.4.
> >>>> Billie
> >>>>> -Joey
> >>>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Joey Echeverria
> >> Director, Federal FTS
> >> Cloudera, Inc.
>
>
>
> --
> Joey Echeverria
> Director, Federal FTS
> Cloudera, Inc.
>
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Joey Echeverria <jo...@cloudera.com>.
I don't think that's a good idea unless you can come up with very
clear version number change.
-Joey
On Fri, Aug 2, 2013 at 2:31 PM, Christopher <ct...@apache.org> wrote:
> Would it be reasonable to consider a version of 1.4 that breaks
> compatibility with 0.20? I'm not really a fan of this, personally, but
> am curious what others think.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Fri, Aug 2, 2013 at 2:22 PM, Joey Echeverria <jo...@cloudera.com> wrote:
>> Sorry for the delay, it's been one of those weeks.
>>
>> The current version would probably not be backwards compatible to
>> 0.20.2 just based on changes in dependencies. We're looking right now
>> to see how hard it is to have three way compatibility (0.20, 1.0,
>> 2.0).
>>
>> -Joey
>>
>> On Thu, Aug 1, 2013 at 7:33 PM, Dave Marion <dl...@comcast.net> wrote:
>>> Any update?
>>>
>>> -----Original Message-----
>>> From: Joey Echeverria [mailto:joey@cloudera.com]
>>> Sent: Monday, July 29, 2013 1:24 PM
>>> To: dev@accumulo.apache.org
>>> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
>>>
>>> We're testing this today. I'll report back what we find.
>>>
>>>
>>> -Joey
>>> —
>>> Sent from Mailbox for iPhone
>>>
>>> On Fri, Jul 26, 2013 at 3:34 PM, null <dl...@comcast.net> wrote:
>>>
>>>> "Will 1.4 still work with 0.20 with these patches?"
>>>> Great point Billie.
>>>> ----- Original Message -----
>>>> From: "Billie Rinaldi" <bi...@gmail.com>
>>>> To: dev@accumulo.apache.org
>>>> Sent: Friday, July 26, 2013 3:02:41 PM
>>>> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch On Fri, Jul
>>>> 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
>>>>> > If these patches are going to be included with 1.4.4 or 1.4.5, I
>>>>> > would
>>>>> like
>>>>> > to see the following test run using CDH4 on at least a 5 node cluster.
>>>>> > More nodes would be better.
>>>>> >
>>>>> > * unit test
>>>>> > * Functional test
>>>>> > * 24 hr Continuous ingest + verification
>>>>> > * 24 hr Continuous ingest + verification + agitation
>>>>> > * 24 hr Random walk
>>>>> > * 24 hr Random walk + agitation
>>>>> >
>>>>> > I may be able to assist with this, but I can not make any promises.
>>>>>
>>>>> Sure thing. Is there already a write-up on running this full battery
>>>>> of tests? I have a 10 node cluster that I can use for this.
>>>>>
>>>>>
>>>>> > Great. I think this would be a good patch for 1.4. I assume that
>>>>> > if a user stays with Hadoop 1 there are no dependency changes?
>>>>>
>>>>> Yup. It works the same way as 1.5 where all of the dependency changes
>>>>> are in a Hadoop 2.0 profile.
>>>>>
>>>> In 1.5.0, we gave up on compatibility with 0.20 (and early versions of
>>>> 1.0) to make the compatibility requirements simpler; we ended up
>>>> without dependency changes in the hadoop version profiles. Will 1.4
>>>> still work with 0.20 with these patches? If there are dependency
>>>> changes in the profiles, 1.4 would have to be compiled against a
>>>> hadoop version compatible with the running version of hadoop, correct?
>>>> We had some trouble in the
>>>> 1.5 release process with figuring out how to provide multiple binary
>>>> artifacts (each compiled against a different version of hadoop) for
>>>> the same release. Just something we should consider before we are in
>>>> the midst of releasing 1.4.4.
>>>> Billie
>>>>> -Joey
>>>>>
>>>
>>
>>
>>
>> --
>> Joey Echeverria
>> Director, Federal FTS
>> Cloudera, Inc.
--
Joey Echeverria
Director, Federal FTS
Cloudera, Inc.
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Christopher <ct...@apache.org>.
Would it be reasonable to consider a version of 1.4 that breaks
compatibility with 0.20? I'm not really a fan of this, personally, but
am curious what others think.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Fri, Aug 2, 2013 at 2:22 PM, Joey Echeverria <jo...@cloudera.com> wrote:
> Sorry for the delay, it's been one of those weeks.
>
> The current version would probably not be backwards compatible to
> 0.20.2 just based on changes in dependencies. We're looking right now
> to see how hard it is to have three way compatibility (0.20, 1.0,
> 2.0).
>
> -Joey
>
> On Thu, Aug 1, 2013 at 7:33 PM, Dave Marion <dl...@comcast.net> wrote:
>> Any update?
>>
>> -----Original Message-----
>> From: Joey Echeverria [mailto:joey@cloudera.com]
>> Sent: Monday, July 29, 2013 1:24 PM
>> To: dev@accumulo.apache.org
>> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
>>
>> We're testing this today. I'll report back what we find.
>>
>>
>> -Joey
>> —
>> Sent from Mailbox for iPhone
>>
>> On Fri, Jul 26, 2013 at 3:34 PM, null <dl...@comcast.net> wrote:
>>
>>> "Will 1.4 still work with 0.20 with these patches?"
>>> Great point Billie.
>>> ----- Original Message -----
>>> From: "Billie Rinaldi" <bi...@gmail.com>
>>> To: dev@accumulo.apache.org
>>> Sent: Friday, July 26, 2013 3:02:41 PM
>>> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch On Fri, Jul
>>> 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
>>>> > If these patches are going to be included with 1.4.4 or 1.4.5, I
>>>> > would
>>>> like
>>>> > to see the following test run using CDH4 on at least a 5 node cluster.
>>>> > More nodes would be better.
>>>> >
>>>> > * unit test
>>>> > * Functional test
>>>> > * 24 hr Continuous ingest + verification
>>>> > * 24 hr Continuous ingest + verification + agitation
>>>> > * 24 hr Random walk
>>>> > * 24 hr Random walk + agitation
>>>> >
>>>> > I may be able to assist with this, but I can not make any promises.
>>>>
>>>> Sure thing. Is there already a write-up on running this full battery
>>>> of tests? I have a 10 node cluster that I can use for this.
>>>>
>>>>
>>>> > Great. I think this would be a good patch for 1.4. I assume that
>>>> > if a user stays with Hadoop 1 there are no dependency changes?
>>>>
>>>> Yup. It works the same way as 1.5 where all of the dependency changes
>>>> are in a Hadoop 2.0 profile.
>>>>
>>> In 1.5.0, we gave up on compatibility with 0.20 (and early versions of
>>> 1.0) to make the compatibility requirements simpler; we ended up
>>> without dependency changes in the hadoop version profiles. Will 1.4
>>> still work with 0.20 with these patches? If there are dependency
>>> changes in the profiles, 1.4 would have to be compiled against a
>>> hadoop version compatible with the running version of hadoop, correct?
>>> We had some trouble in the
>>> 1.5 release process with figuring out how to provide multiple binary
>>> artifacts (each compiled against a different version of hadoop) for
>>> the same release. Just something we should consider before we are in
>>> the midst of releasing 1.4.4.
>>> Billie
>>>> -Joey
>>>>
>>
>
>
>
> --
> Joey Echeverria
> Director, Federal FTS
> Cloudera, Inc.
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Joey Echeverria <jo...@cloudera.com>.
Sorry for the delay, it's been one of those weeks.
The current version would probably not be backwards compatible to
0.20.2 just based on changes in dependencies. We're looking right now
to see how hard it is to have three way compatibility (0.20, 1.0,
2.0).
-Joey
On Thu, Aug 1, 2013 at 7:33 PM, Dave Marion <dl...@comcast.net> wrote:
> Any update?
>
> -----Original Message-----
> From: Joey Echeverria [mailto:joey@cloudera.com]
> Sent: Monday, July 29, 2013 1:24 PM
> To: dev@accumulo.apache.org
> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
>
> We're testing this today. I'll report back what we find.
>
>
> -Joey
> —
> Sent from Mailbox for iPhone
>
> On Fri, Jul 26, 2013 at 3:34 PM, null <dl...@comcast.net> wrote:
>
>> "Will 1.4 still work with 0.20 with these patches?"
>> Great point Billie.
>> ----- Original Message -----
>> From: "Billie Rinaldi" <bi...@gmail.com>
>> To: dev@accumulo.apache.org
>> Sent: Friday, July 26, 2013 3:02:41 PM
>> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch On Fri, Jul
>> 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
>>> > If these patches are going to be included with 1.4.4 or 1.4.5, I
>>> > would
>>> like
>>> > to see the following test run using CDH4 on at least a 5 node cluster.
>>> > More nodes would be better.
>>> >
>>> > * unit test
>>> > * Functional test
>>> > * 24 hr Continuous ingest + verification
>>> > * 24 hr Continuous ingest + verification + agitation
>>> > * 24 hr Random walk
>>> > * 24 hr Random walk + agitation
>>> >
>>> > I may be able to assist with this, but I can not make any promises.
>>>
>>> Sure thing. Is there already a write-up on running this full battery
>>> of tests? I have a 10 node cluster that I can use for this.
>>>
>>>
>>> > Great. I think this would be a good patch for 1.4. I assume that
>>> > if a user stays with Hadoop 1 there are no dependency changes?
>>>
>>> Yup. It works the same way as 1.5 where all of the dependency changes
>>> are in a Hadoop 2.0 profile.
>>>
>> In 1.5.0, we gave up on compatibility with 0.20 (and early versions of
>> 1.0) to make the compatibility requirements simpler; we ended up
>> without dependency changes in the hadoop version profiles. Will 1.4
>> still work with 0.20 with these patches? If there are dependency
>> changes in the profiles, 1.4 would have to be compiled against a
>> hadoop version compatible with the running version of hadoop, correct?
>> We had some trouble in the
>> 1.5 release process with figuring out how to provide multiple binary
>> artifacts (each compiled against a different version of hadoop) for
>> the same release. Just something we should consider before we are in
>> the midst of releasing 1.4.4.
>> Billie
>>> -Joey
>>>
>
--
Joey Echeverria
Director, Federal FTS
Cloudera, Inc.
RE: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Dave Marion <dl...@comcast.net>.
Any update?
-----Original Message-----
From: Joey Echeverria [mailto:joey@cloudera.com]
Sent: Monday, July 29, 2013 1:24 PM
To: dev@accumulo.apache.org
Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
We're testing this today. I'll report back what we find.
-Joey
—
Sent from Mailbox for iPhone
On Fri, Jul 26, 2013 at 3:34 PM, null <dl...@comcast.net> wrote:
> "Will 1.4 still work with 0.20 with these patches?"
> Great point Billie.
> ----- Original Message -----
> From: "Billie Rinaldi" <bi...@gmail.com>
> To: dev@accumulo.apache.org
> Sent: Friday, July 26, 2013 3:02:41 PM
> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch On Fri, Jul
> 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
>> > If these patches are going to be included with 1.4.4 or 1.4.5, I
>> > would
>> like
>> > to see the following test run using CDH4 on at least a 5 node cluster.
>> > More nodes would be better.
>> >
>> > * unit test
>> > * Functional test
>> > * 24 hr Continuous ingest + verification
>> > * 24 hr Continuous ingest + verification + agitation
>> > * 24 hr Random walk
>> > * 24 hr Random walk + agitation
>> >
>> > I may be able to assist with this, but I can not make any promises.
>>
>> Sure thing. Is there already a write-up on running this full battery
>> of tests? I have a 10 node cluster that I can use for this.
>>
>>
>> > Great. I think this would be a good patch for 1.4. I assume that
>> > if a user stays with Hadoop 1 there are no dependency changes?
>>
>> Yup. It works the same way as 1.5 where all of the dependency changes
>> are in a Hadoop 2.0 profile.
>>
> In 1.5.0, we gave up on compatibility with 0.20 (and early versions of
> 1.0) to make the compatibility requirements simpler; we ended up
> without dependency changes in the hadoop version profiles. Will 1.4
> still work with 0.20 with these patches? If there are dependency
> changes in the profiles, 1.4 would have to be compiled against a
> hadoop version compatible with the running version of hadoop, correct?
> We had some trouble in the
> 1.5 release process with figuring out how to provide multiple binary
> artifacts (each compiled against a different version of hadoop) for
> the same release. Just something we should consider before we are in
> the midst of releasing 1.4.4.
> Billie
>> -Joey
>>
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Joey Echeverria <jo...@cloudera.com>.
We're testing this today. I'll report back what we find.
-Joey
—
Sent from Mailbox for iPhone
On Fri, Jul 26, 2013 at 3:34 PM, null <dl...@comcast.net> wrote:
> "Will 1.4 still work with 0.20 with these patches?"
> Great point Billie.
> ----- Original Message -----
> From: "Billie Rinaldi" <bi...@gmail.com>
> To: dev@accumulo.apache.org
> Sent: Friday, July 26, 2013 3:02:41 PM
> Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
> On Fri, Jul 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
>> > If these patches are going to be included with 1.4.4 or 1.4.5, I would
>> like
>> > to see the following test run using CDH4 on at least a 5 node cluster.
>> > More nodes would be better.
>> >
>> > * unit test
>> > * Functional test
>> > * 24 hr Continuous ingest + verification
>> > * 24 hr Continuous ingest + verification + agitation
>> > * 24 hr Random walk
>> > * 24 hr Random walk + agitation
>> >
>> > I may be able to assist with this, but I can not make any promises.
>>
>> Sure thing. Is there already a write-up on running this full battery
>> of tests? I have a 10 node cluster that I can use for this.
>>
>>
>> > Great. I think this would be a good patch for 1.4. I assume that if a
>> > user stays with Hadoop 1 there are no dependency changes?
>>
>> Yup. It works the same way as 1.5 where all of the dependency changes
>> are in a Hadoop 2.0 profile.
>>
> In 1.5.0, we gave up on compatibility with 0.20 (and early versions of 1.0)
> to make the compatibility requirements simpler; we ended up without
> dependency changes in the hadoop version profiles. Will 1.4 still work
> with 0.20 with these patches? If there are dependency changes in the
> profiles, 1.4 would have to be compiled against a hadoop version compatible
> with the running version of hadoop, correct? We had some trouble in the
> 1.5 release process with figuring out how to provide multiple binary
> artifacts (each compiled against a different version of hadoop) for the
> same release. Just something we should consider before we are in the midst
> of releasing 1.4.4.
> Billie
>> -Joey
>>
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by dl...@comcast.net.
"Will 1.4 still work with 0.20 with these patches?"
Great point Billie.
----- Original Message -----
From: "Billie Rinaldi" <bi...@gmail.com>
To: dev@accumulo.apache.org
Sent: Friday, July 26, 2013 3:02:41 PM
Subject: Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
On Fri, Jul 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
> > If these patches are going to be included with 1.4.4 or 1.4.5, I would
> like
> > to see the following test run using CDH4 on at least a 5 node cluster.
> > More nodes would be better.
> >
> > * unit test
> > * Functional test
> > * 24 hr Continuous ingest + verification
> > * 24 hr Continuous ingest + verification + agitation
> > * 24 hr Random walk
> > * 24 hr Random walk + agitation
> >
> > I may be able to assist with this, but I can not make any promises.
>
> Sure thing. Is there already a write-up on running this full battery
> of tests? I have a 10 node cluster that I can use for this.
>
>
> > Great. I think this would be a good patch for 1.4. I assume that if a
> > user stays with Hadoop 1 there are no dependency changes?
>
> Yup. It works the same way as 1.5 where all of the dependency changes
> are in a Hadoop 2.0 profile.
>
In 1.5.0, we gave up on compatibility with 0.20 (and early versions of 1.0)
to make the compatibility requirements simpler; we ended up without
dependency changes in the hadoop version profiles. Will 1.4 still work
with 0.20 with these patches? If there are dependency changes in the
profiles, 1.4 would have to be compiled against a hadoop version compatible
with the running version of hadoop, correct? We had some trouble in the
1.5 release process with figuring out how to provide multiple binary
artifacts (each compiled against a different version of hadoop) for the
same release. Just something we should consider before we are in the midst
of releasing 1.4.4.
Billie
> -Joey
>
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Billie Rinaldi <bi...@gmail.com>.
On Fri, Jul 26, 2013 at 11:33 AM, Joey Echeverria <jo...@cloudera.com> wrote:
> > If these patches are going to be included with 1.4.4 or 1.4.5, I would
> like
> > to see the following test run using CDH4 on at least a 5 node cluster.
> > More nodes would be better.
> >
> > * unit test
> > * Functional test
> > * 24 hr Continuous ingest + verification
> > * 24 hr Continuous ingest + verification + agitation
> > * 24 hr Random walk
> > * 24 hr Random walk + agitation
> >
> > I may be able to assist with this, but I can not make any promises.
>
> Sure thing. Is there already a write-up on running this full battery
> of tests? I have a 10 node cluster that I can use for this.
>
>
> > Great. I think this would be a good patch for 1.4. I assume that if a
> > user stays with Hadoop 1 there are no dependency changes?
>
> Yup. It works the same way as 1.5 where all of the dependency changes
> are in a Hadoop 2.0 profile.
>
In 1.5.0, we gave up on compatibility with 0.20 (and early versions of 1.0)
to make the compatibility requirements simpler; we ended up without
dependency changes in the hadoop version profiles. Will 1.4 still work
with 0.20 with these patches? If there are dependency changes in the
profiles, 1.4 would have to be compiled against a hadoop version compatible
with the running version of hadoop, correct? We had some trouble in the
1.5 release process with figuring out how to provide multiple binary
artifacts (each compiled against a different version of hadoop) for the
same release. Just something we should consider before we are in the midst
of releasing 1.4.4.
Billie
> -Joey
>
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Keith Turner <ke...@deenlo.com>.
On Fri, Jul 26, 2013 at 2:33 PM, Joey Echeverria <jo...@cloudera.com> wrote:
> > If these patches are going to be included with 1.4.4 or 1.4.5, I would
> like
> > to see the following test run using CDH4 on at least a 5 node cluster.
> > More nodes would be better.
> >
> > * unit test
> > * Functional test
> > * 24 hr Continuous ingest + verification
> > * 24 hr Continuous ingest + verification + agitation
> > * 24 hr Random walk
> > * 24 hr Random walk + agitation
> >
> > I may be able to assist with this, but I can not make any promises.
>
> Sure thing. Is there already a write-up on running this full battery
> of tests? I have a 10 node cluster that I can use for this.
>
There are some instructions.
test/system/continuous/README
test/system/randomwalk/README
Continuous ingest has a lot of options. For release testing we do
something like the following.
#configure may need to adjust max mappers and max reducers to make map
reduce job run faster
start-ingest.sh
start-walker.sh
#sleep 24hr
stop-ingest.sh
stop-walker.sh
run-verify.sh
There continuous dir has scripts for starting and stopping the agitator.
We also use this script to agitate while running random walk test.
For random walk we use the All.xml graph, configure it to log errors to
NFS, and run a walker on each node. We look in NFS for walkers that died
or got stuck. The random walk framework will log a message if a node in
the graph gets stuck. It will also log a message when it gets unstuck.
>
> > Great. I think this would be a good patch for 1.4. I assume that if a
> > user stays with Hadoop 1 there are no dependency changes?
>
> Yup. It works the same way as 1.5 where all of the dependency changes
> are in a Hadoop 2.0 profile.
>
> -Joey
>
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Joey Echeverria <jo...@cloudera.com>.
> If these patches are going to be included with 1.4.4 or 1.4.5, I would like
> to see the following test run using CDH4 on at least a 5 node cluster.
> More nodes would be better.
>
> * unit test
> * Functional test
> * 24 hr Continuous ingest + verification
> * 24 hr Continuous ingest + verification + agitation
> * 24 hr Random walk
> * 24 hr Random walk + agitation
>
> I may be able to assist with this, but I can not make any promises.
Sure thing. Is there already a write-up on running this full battery
of tests? I have a 10 node cluster that I can use for this.
> Great. I think this would be a good patch for 1.4. I assume that if a
> user stays with Hadoop 1 there are no dependency changes?
Yup. It works the same way as 1.5 where all of the dependency changes
are in a Hadoop 2.0 profile.
-Joey
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Keith Turner <ke...@deenlo.com>.
On Fri, Jul 26, 2013 at 12:24 PM, Joey Echeverria <jo...@cloudera.com> wrote:
> We have both the unit tests and the full system test suite hooked up to a
> Jenkins build server.
>
If these patches are going to be included with 1.4.4 or 1.4.5, I would like
to see the following test run using CDH4 on at least a 5 node cluster.
More nodes would be better.
* unit test
* Functional test
* 24 hr Continuous ingest + verification
* 24 hr Continuous ingest + verification + agitation
* 24 hr Random walk
* 24 hr Random walk + agitation
I may be able to assist with this, but I can not make any promises.
>
> There are still a couple of tests that fail periodically with the full
> system test due to timeouts. We're working on those which is why our
> current release is just a beta.
>
> There are no API changes or Accumulo behavior changes. You can use
> unmodified 1.4.x clients with our release of the server daemons.
>
Great. I think this would be a good patch for 1.4. I assume that if a
user stays with Hadoop 1 there are no dependency changes?
>
> -Joey
>
>
> On Fri, Jul 26, 2013 at 11:45 AM, Keith Turner <ke...@deenlo.com> wrote:
>
> > On Fri, Jul 26, 2013 at 11:02 AM, Joey Echeverria <jo...@cloudera.com>
> > wrote:
> >
> > > Cloudera announced last night our support for Accumulo 1.4.3 on CDH4:
> > >
> > > http://www.slideshare.net/JoeyEcheverria/apache-accumulo-and-cloudera
> > >
> > > This required back porting about 11 patches in whole or in part from
> the
> > > 1.5 line on top of 1.4.3. Our release is still in a semi-private beta,
> > but
> > > when it's fully public it will be downloadable along with all of the
> > extra
> > > patches that we committed.
> > >
> > > My question is if the community would be interested in us pulling those
> > > back ports upstream?
> > >
> >
> > What testing has been done? It would be nice to run accumulo's full test
> > suite against 1.4.3+CDH4.
> >
> > Are there any Accumulo API changes or Accumulo behavior changes?
> >
> >
> > > I believe this would violate the previously agreed upon rule of no
> > feature
> > > back ports to 1.4.3, depending on how we "label" support for Hadoop
> 2.0.
> >
> >
> > > Thoughts?
> > >
> > > -Joey
> > >
> >
>
>
>
> --
> Joey Echeverria
> Director, Federal FTS
> Cloudera, Inc.
>
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Joey Echeverria <jo...@cloudera.com>.
We have both the unit tests and the full system test suite hooked up to a
Jenkins build server.
There are still a couple of tests that fail periodically with the full
system test due to timeouts. We're working on those which is why our
current release is just a beta.
There are no API changes or Accumulo behavior changes. You can use
unmodified 1.4.x clients with our release of the server daemons.
-Joey
On Fri, Jul 26, 2013 at 11:45 AM, Keith Turner <ke...@deenlo.com> wrote:
> On Fri, Jul 26, 2013 at 11:02 AM, Joey Echeverria <jo...@cloudera.com>
> wrote:
>
> > Cloudera announced last night our support for Accumulo 1.4.3 on CDH4:
> >
> > http://www.slideshare.net/JoeyEcheverria/apache-accumulo-and-cloudera
> >
> > This required back porting about 11 patches in whole or in part from the
> > 1.5 line on top of 1.4.3. Our release is still in a semi-private beta,
> but
> > when it's fully public it will be downloadable along with all of the
> extra
> > patches that we committed.
> >
> > My question is if the community would be interested in us pulling those
> > back ports upstream?
> >
>
> What testing has been done? It would be nice to run accumulo's full test
> suite against 1.4.3+CDH4.
>
> Are there any Accumulo API changes or Accumulo behavior changes?
>
>
> > I believe this would violate the previously agreed upon rule of no
> feature
> > back ports to 1.4.3, depending on how we "label" support for Hadoop 2.0.
>
>
> > Thoughts?
> >
> > -Joey
> >
>
--
Joey Echeverria
Director, Federal FTS
Cloudera, Inc.
Re: Hadoop 2.0 Support for Accumulo 1.4 Branch
Posted by Keith Turner <ke...@deenlo.com>.
On Fri, Jul 26, 2013 at 11:02 AM, Joey Echeverria <jo...@cloudera.com> wrote:
> Cloudera announced last night our support for Accumulo 1.4.3 on CDH4:
>
> http://www.slideshare.net/JoeyEcheverria/apache-accumulo-and-cloudera
>
> This required back porting about 11 patches in whole or in part from the
> 1.5 line on top of 1.4.3. Our release is still in a semi-private beta, but
> when it's fully public it will be downloadable along with all of the extra
> patches that we committed.
>
> My question is if the community would be interested in us pulling those
> back ports upstream?
>
What testing has been done? It would be nice to run accumulo's full test
suite against 1.4.3+CDH4.
Are there any Accumulo API changes or Accumulo behavior changes?
> I believe this would violate the previously agreed upon rule of no feature
> back ports to 1.4.3, depending on how we "label" support for Hadoop 2.0.
> Thoughts?
>
> -Joey
>