You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by John Vines <vi...@apache.org> on 2014/10/07 00:38:09 UTC
1.7 release timeline
Moving this to it's own thread...
On Mon, Oct 6, 2014 at 5:54 PM, Mike Drob <ma...@cloudera.com> wrote:
> Related: Do we have a release timeline for 1.7?
>
> On Mon, Oct 6, 2014 at 4:51 PM, Christopher <ct...@apache.org> wrote:
>
> > On Mon, Oct 6, 2014 at 5:20 PM, Sean Busbey <bu...@cloudera.com> wrote:
> >
> > > On Mon, Oct 6, 2014 at 4:12 PM, Mike Drob <ma...@cloudera.com> wrote:
> > >
> > > >
> > > >
> > > > In general, I'm inclined to leave as much in as possible, and then if
> > we
> > > > must remove things then do so in 2.0.0. I know that our compatibility
> > > > statement only promises one minor version, but that doesn't mean we
> > have
> > > to
> > > > be strict at every opportunity.
> > > >
> > > > Mike
> > > >
> > > >
> > >
> > > Related, I'd like to EOL 1.5 shortly after 1.7 gets released. I don't
> > want
> > > to derail this thread with that discussion, but my guess is it's a much
> > > easier sell if we're conservative about removing things. Just so
> everyone
> > > knows where I'm coming from.
> > >
> > >
> > >
> > (+1 for EOL 1.5 after)
> >
> > In general, does this mean that you're okay with removing stuff
> deprecated
> > prior to 1.5? With the exception of the instance.getConfiguration stuff,
> > which was deprecated in 1.6.0 and I'd like to remove in 1.7.0, due to its
> > problematic nature (requires further discussion), I could restrict the
> > remaining cleanup to only stuff deprecated prior to 1.5.
> >
>
Re: 1.7 release timeline
Posted by Josh Elser <jo...@gmail.com>.
Oh, please, no worries. Part of figuring out the timeline for certain is
to figure out what we plan to accomplish!
Happy to see productive conversation on the subject.
Christopher wrote:
> Ah, I assumed it had instrumentation! Okay, cool. Thanks for the
> clarification. So, I see that it definitely makes sense to work towards
> merging htrace and cloudtrace/accumulo-trace, and fix the shortcomings as
> it matures. (Now, back to the show/main thread. ;) )
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Mon, Oct 6, 2014 at 9:36 PM, Billie Rinaldi<bi...@gmail.com>
> wrote:
>
>> Zipkin is a possible replacement for our trace collection system. It does
>> not provide instrumentation like cloudtrace or htrace, so even if we make
>> zipkin the default collection system we will still need instrumentation.
>> Anyway, we can discuss the details and approach elsewhere. I'd certainly
>> want the trace work to be in 2.0, but if we decide not to put it in 1.7
>> that would be okay.
>>
>> On Mon, Oct 6, 2014 at 5:59 PM, Christopher<ct...@apache.org> wrote:
>>
>>> Would replacing cloudtrace be part of 1.7? I'm not sure about that. I'd
>>> like to see where that's headed before we decide on that. Personally, I'd
>>> prefer Zipkin, since htrace is basically a copy of
>>> cloudtrace/accumulo-trace, and it has some of the same issues (millis
>> time,
>>> for instance, instead of relative nanos, which is independent of the
>> system
>>> clock and actually intended for time spans).
>>>
>>> I think the upgrade guarantees are more a 2.0.0 thing, but I think we can
>>> be a bit more conservative in 1.x to move towards that. I wouldn't mind
>>> dropping Hadoop 1 support in 1.7.0. (I guess we should just vote on
>> that).
>>> I'd really like to include the VolumeChooser improvements (in particular
>>> ACCUMULO-3177, which depends on ACCUMULO-3176).
>>>
>>>
>>> --
>>> Christopher L Tubbs II
>>> http://gravatar.com/ctubbsii
>>>
>>> On Mon, Oct 6, 2014 at 8:38 PM, Josh Elser<jo...@gmail.com> wrote:
>>>
>>>> Thanks, John.
>>>>
>>>> I was thinking about trying to gun for January time-frame for a
>> release.
>>>> I'd love to say before 2014 is over, but that probably just won't
>> happen
>>>> for a major release with the holidays.
>>>>
>>>> For 1.7 right now, I see the following "bigger" items (correct me where
>>>> I'm wrong):
>>>>
>>>> * Replication (done)
>>>> * Upgrade rules/guarantees (proposed)
>>>> * Replace cloudtrace (in-progress)
>>>> * Rewrite monitor, include REST service (in-progress)
>>>> * Drop Hadoop 1 support (proposed)
>>>> * Decouple MiniAccumulo from ITs (in-progress)
>>>> * Other minicluster types: in-process, shim to real instance
>>> (in-progress)
>>>> * Support Hadoop metrics2 (proposed)
>>>> * A few WAL/metadata related performance improvements (in-progress)
>>>>
>>>> Also, would be good to check the In-Progress state issues on JIRA. What
>>> do
>>>> people think?
>>>>
>>>>
>>>> John Vines wrote:
>>>>
>>>>> Moving this to it's own thread...
>>>>>
>>>>> On Mon, Oct 6, 2014 at 5:54 PM, Mike Drob<ma...@cloudera.com>
>> wrote:
>>>>> Related: Do we have a release timeline for 1.7?
>>>>>>
>
Re: 1.7 release timeline
Posted by Christopher <ct...@apache.org>.
Ah, I assumed it had instrumentation! Okay, cool. Thanks for the
clarification. So, I see that it definitely makes sense to work towards
merging htrace and cloudtrace/accumulo-trace, and fix the shortcomings as
it matures. (Now, back to the show/main thread. ;) )
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Mon, Oct 6, 2014 at 9:36 PM, Billie Rinaldi <bi...@gmail.com>
wrote:
> Zipkin is a possible replacement for our trace collection system. It does
> not provide instrumentation like cloudtrace or htrace, so even if we make
> zipkin the default collection system we will still need instrumentation.
> Anyway, we can discuss the details and approach elsewhere. I'd certainly
> want the trace work to be in 2.0, but if we decide not to put it in 1.7
> that would be okay.
>
> On Mon, Oct 6, 2014 at 5:59 PM, Christopher <ct...@apache.org> wrote:
>
> > Would replacing cloudtrace be part of 1.7? I'm not sure about that. I'd
> > like to see where that's headed before we decide on that. Personally, I'd
> > prefer Zipkin, since htrace is basically a copy of
> > cloudtrace/accumulo-trace, and it has some of the same issues (millis
> time,
> > for instance, instead of relative nanos, which is independent of the
> system
> > clock and actually intended for time spans).
> >
> > I think the upgrade guarantees are more a 2.0.0 thing, but I think we can
> > be a bit more conservative in 1.x to move towards that. I wouldn't mind
> > dropping Hadoop 1 support in 1.7.0. (I guess we should just vote on
> that).
> >
> > I'd really like to include the VolumeChooser improvements (in particular
> > ACCUMULO-3177, which depends on ACCUMULO-3176).
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> > On Mon, Oct 6, 2014 at 8:38 PM, Josh Elser <jo...@gmail.com> wrote:
> >
> > > Thanks, John.
> > >
> > > I was thinking about trying to gun for January time-frame for a
> release.
> > > I'd love to say before 2014 is over, but that probably just won't
> happen
> > > for a major release with the holidays.
> > >
> > > For 1.7 right now, I see the following "bigger" items (correct me where
> > > I'm wrong):
> > >
> > > * Replication (done)
> > > * Upgrade rules/guarantees (proposed)
> > > * Replace cloudtrace (in-progress)
> > > * Rewrite monitor, include REST service (in-progress)
> > > * Drop Hadoop 1 support (proposed)
> > > * Decouple MiniAccumulo from ITs (in-progress)
> > > * Other minicluster types: in-process, shim to real instance
> > (in-progress)
> > > * Support Hadoop metrics2 (proposed)
> > > * A few WAL/metadata related performance improvements (in-progress)
> > >
> > > Also, would be good to check the In-Progress state issues on JIRA. What
> > do
> > > people think?
> > >
> > >
> > > John Vines wrote:
> > >
> > >> Moving this to it's own thread...
> > >>
> > >> On Mon, Oct 6, 2014 at 5:54 PM, Mike Drob<ma...@cloudera.com>
> wrote:
> > >>
> > >> Related: Do we have a release timeline for 1.7?
> > >>>
> > >>>
> >
>
Re: 1.7 release timeline
Posted by Billie Rinaldi <bi...@gmail.com>.
Zipkin is a possible replacement for our trace collection system. It does
not provide instrumentation like cloudtrace or htrace, so even if we make
zipkin the default collection system we will still need instrumentation.
Anyway, we can discuss the details and approach elsewhere. I'd certainly
want the trace work to be in 2.0, but if we decide not to put it in 1.7
that would be okay.
On Mon, Oct 6, 2014 at 5:59 PM, Christopher <ct...@apache.org> wrote:
> Would replacing cloudtrace be part of 1.7? I'm not sure about that. I'd
> like to see where that's headed before we decide on that. Personally, I'd
> prefer Zipkin, since htrace is basically a copy of
> cloudtrace/accumulo-trace, and it has some of the same issues (millis time,
> for instance, instead of relative nanos, which is independent of the system
> clock and actually intended for time spans).
>
> I think the upgrade guarantees are more a 2.0.0 thing, but I think we can
> be a bit more conservative in 1.x to move towards that. I wouldn't mind
> dropping Hadoop 1 support in 1.7.0. (I guess we should just vote on that).
>
> I'd really like to include the VolumeChooser improvements (in particular
> ACCUMULO-3177, which depends on ACCUMULO-3176).
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Mon, Oct 6, 2014 at 8:38 PM, Josh Elser <jo...@gmail.com> wrote:
>
> > Thanks, John.
> >
> > I was thinking about trying to gun for January time-frame for a release.
> > I'd love to say before 2014 is over, but that probably just won't happen
> > for a major release with the holidays.
> >
> > For 1.7 right now, I see the following "bigger" items (correct me where
> > I'm wrong):
> >
> > * Replication (done)
> > * Upgrade rules/guarantees (proposed)
> > * Replace cloudtrace (in-progress)
> > * Rewrite monitor, include REST service (in-progress)
> > * Drop Hadoop 1 support (proposed)
> > * Decouple MiniAccumulo from ITs (in-progress)
> > * Other minicluster types: in-process, shim to real instance
> (in-progress)
> > * Support Hadoop metrics2 (proposed)
> > * A few WAL/metadata related performance improvements (in-progress)
> >
> > Also, would be good to check the In-Progress state issues on JIRA. What
> do
> > people think?
> >
> >
> > John Vines wrote:
> >
> >> Moving this to it's own thread...
> >>
> >> On Mon, Oct 6, 2014 at 5:54 PM, Mike Drob<ma...@cloudera.com> wrote:
> >>
> >> Related: Do we have a release timeline for 1.7?
> >>>
> >>>
>
Re: 1.7 release timeline
Posted by Josh Elser <jo...@gmail.com>.
Yes, of course. We definitely need to see some code here before it gets
officially "slated" for 1.7. I just know that efforts are being put
towards it, so I wanted to list it.
Christopher wrote:
> Would replacing cloudtrace be part of1.7? I'm not sure about that. I'd
> like to see where that's headed before we decide on that. Personally, I'd
> prefer Zipkin, since htrace is basically a copy of
> cloudtrace/accumulo-trace, and it has some of the same issues (millis time,
> for instance, instead of relative nanos, which is independent of the system
> clock and actually intended for time spans).
Re: 1.7 release timeline
Posted by Christopher <ct...@apache.org>.
Would replacing cloudtrace be part of 1.7? I'm not sure about that. I'd
like to see where that's headed before we decide on that. Personally, I'd
prefer Zipkin, since htrace is basically a copy of
cloudtrace/accumulo-trace, and it has some of the same issues (millis time,
for instance, instead of relative nanos, which is independent of the system
clock and actually intended for time spans).
I think the upgrade guarantees are more a 2.0.0 thing, but I think we can
be a bit more conservative in 1.x to move towards that. I wouldn't mind
dropping Hadoop 1 support in 1.7.0. (I guess we should just vote on that).
I'd really like to include the VolumeChooser improvements (in particular
ACCUMULO-3177, which depends on ACCUMULO-3176).
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Mon, Oct 6, 2014 at 8:38 PM, Josh Elser <jo...@gmail.com> wrote:
> Thanks, John.
>
> I was thinking about trying to gun for January time-frame for a release.
> I'd love to say before 2014 is over, but that probably just won't happen
> for a major release with the holidays.
>
> For 1.7 right now, I see the following "bigger" items (correct me where
> I'm wrong):
>
> * Replication (done)
> * Upgrade rules/guarantees (proposed)
> * Replace cloudtrace (in-progress)
> * Rewrite monitor, include REST service (in-progress)
> * Drop Hadoop 1 support (proposed)
> * Decouple MiniAccumulo from ITs (in-progress)
> * Other minicluster types: in-process, shim to real instance (in-progress)
> * Support Hadoop metrics2 (proposed)
> * A few WAL/metadata related performance improvements (in-progress)
>
> Also, would be good to check the In-Progress state issues on JIRA. What do
> people think?
>
>
> John Vines wrote:
>
>> Moving this to it's own thread...
>>
>> On Mon, Oct 6, 2014 at 5:54 PM, Mike Drob<ma...@cloudera.com> wrote:
>>
>> Related: Do we have a release timeline for 1.7?
>>>
>>>
Re: 1.7 release timeline
Posted by Christopher <ct...@apache.org>.
On Wed, Oct 8, 2014 at 10:44 AM, Josh Elser <jo...@gmail.com> wrote:
> Sean Busbey wrote:
>
>> On Wed, Oct 8, 2014 at 12:13 AM, Josh Elser<jo...@gmail.com> wrote:
>>
>> Forgot one:
>>>
>>> *Drop Hadoop 1 support*
>>> - We would no longer care about maintaining Hadoop 1 APIs (get rid of
>>> crappy reflection)
>>> - 2.2.0 (Hadoop 2 "stable") came out just under 1 year ago
>>> - Can be done for 1.7 or reconsidered for 2.0
>>>
>>>
>>>
>>> Do we already know if focusing on just Hadoop 2.2.0+ support will
>> result in
>> any API impact?
>>
>
> I can't think of anything off the top of my head. Most of the things that
> I was thinking of are buried in impl or server-side.
>
> *Upgrade test script*
>>>
>>>> - Keith mentioned there's some code from John McNamee that might help
>>>> testing upgrade paths
>>>>
>>>>
>>>> If we can make this a contrib, I have some code from Mike and I that
>> we'd
>> like to add.
>>
>
> I'm still not sure where exactly this code is that Keith mentioned (or
> what it does), but please feel free to make a plan. It would be nice to get
> a roadmap for upgrade support/guarantees/testing.
>
The code I believe we were referring to was attached as a patch to:
https://issues.apache.org/jira/browse/ACCUMULO-2145
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
Re: 1.7 release timeline
Posted by Josh Elser <jo...@gmail.com>.
Sean Busbey wrote:
> On Wed, Oct 8, 2014 at 12:13 AM, Josh Elser<jo...@gmail.com> wrote:
>
>> Forgot one:
>>
>> *Drop Hadoop 1 support*
>> - We would no longer care about maintaining Hadoop 1 APIs (get rid of
>> crappy reflection)
>> - 2.2.0 (Hadoop 2 "stable") came out just under 1 year ago
>> - Can be done for 1.7 or reconsidered for 2.0
>>
>>
>>
> Do we already know if focusing on just Hadoop 2.2.0+ support will result in
> any API impact?
I can't think of anything off the top of my head. Most of the things
that I was thinking of are buried in impl or server-side.
>> *Upgrade test script*
>>> - Keith mentioned there's some code from John McNamee that might help
>>> testing upgrade paths
>>>
>>>
> If we can make this a contrib, I have some code from Mike and I that we'd
> like to add.
I'm still not sure where exactly this code is that Keith mentioned (or
what it does), but please feel free to make a plan. It would be nice to
get a roadmap for upgrade support/guarantees/testing.
Re: 1.7 release timeline
Posted by Josh Elser <jo...@gmail.com>.
Aggression being "do we go out of our way to prevent Hadoop 1 from
working". Like you said, it works now and we have nothing in the code
that doesn't reasonably support Hadoop 1 and 2.
I agree with reconsidering this again when 2.0.0 circles around once more.
Thanks for everyone's input.
Christopher wrote:
> I don't think it's about aggression... it's about progress. We can continue
> to support a version with bugfixes that does work on Hadoop 1 for some
> time. If it's likely to cause problems in the bleeding edge, though... we
> can drop it there. In any case, I'm fine with doing this in 2.0.0, instead,
> if it makes it easier to leave a support window in 1.x for Hadoop 1.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Wed, Oct 8, 2014 at 12:43 PM, Josh Elser<jo...@gmail.com> wrote:
>
>> Sean Busbey wrote:
>>
>>> On Wed, Oct 8, 2014 at 11:12 AM, Christopher<ct...@apache.org> wrote:
>>>
>>> On Wed, Oct 8, 2014 at 9:47 AM, Sean Busbey<bu...@cloudera.com> wrote:
>>>> On Wed, Oct 8, 2014 at 12:13 AM, Josh Elser<jo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Forgot one:
>>>>>> *Drop Hadoop 1 support*
>>>>>> - We would no longer care about maintaining Hadoop 1 APIs (get rid
>>>>>> of
>>>>>> crappy reflection)
>>>>>> - 2.2.0 (Hadoop 2 "stable") came out just under 1 year ago
>>>>>> - Can be done for 1.7 or reconsidered for 2.0
>>>>>>
>>>>>>
>>>>>>
>>>>>> Do we already know if focusing on just Hadoop 2.2.0+ support will
>>>>> result
>>>>>
>>>> in
>>>>
>>>>> any API impact?
>>>>>
>>>>>
>>>>> I can't imagine it will impact our client API much, if any, but it'll
>>>> certainly help simplify and fix bugs in our use of Hadoop (when they
>>>> appear), and may result in fewer compatibility issues in MapReduce and
>>>> elsewhere (dynamic class loading, reflection workarounds). It should also
>>>> allow us to provide some guarantees in durability-related features
>>>> (because
>>>> we'll know they exist). It should also help simplify documentation and
>>>> example configs, as well as reduce testing burdens.
>>>>
>>>> +1 to dropping Hadoop<2.2.0 in 1.7.0.
>>>>
>>>>
>>>>
>>> If it doesn't impact API or usage I'd also be +1. If it ends up leaking
>>> out
>>> somewhere, I'd want to take the opportunity of 1.7.0 to flag deprecations
>>> so the transition can be made less severe (and remove in 2.0).
>>>
>>> Frankly, given the general expectations on durability we formed in 1.4 I'd
>>> say it's dangerous for people to be running 1.5+ on Hadoop 1.
>>>
>>>
>> Yeah, that's definitely what has to be weighed. Are we so aggressive to
>> prevent users from even running on Hadoop 1 because it is inherently
>> missing some of the sync mechanics needed to ensure no data loss regardless
>> of redundant power?
>>
>> We warn users when they don't have the necessary options set at the HDFS
>> level to properly sync, but we don't keep them from running like that.
>>
>> The more I think about it, I think I'd be more in favor of continuing to
>> strongly encourage users to adopt Hadoop 2 and revisit dropping Hadoop 1
>> for 2.0.0.
>>
>
Re: 1.7 release timeline
Posted by Christopher <ct...@apache.org>.
I don't think it's about aggression... it's about progress. We can continue
to support a version with bugfixes that does work on Hadoop 1 for some
time. If it's likely to cause problems in the bleeding edge, though... we
can drop it there. In any case, I'm fine with doing this in 2.0.0, instead,
if it makes it easier to leave a support window in 1.x for Hadoop 1.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Wed, Oct 8, 2014 at 12:43 PM, Josh Elser <jo...@gmail.com> wrote:
> Sean Busbey wrote:
>
>> On Wed, Oct 8, 2014 at 11:12 AM, Christopher<ct...@apache.org> wrote:
>>
>> On Wed, Oct 8, 2014 at 9:47 AM, Sean Busbey<bu...@cloudera.com> wrote:
>>>
>>> On Wed, Oct 8, 2014 at 12:13 AM, Josh Elser<jo...@gmail.com>
>>>>
>>> wrote:
>>>
>>>> Forgot one:
>>>>>
>>>>> *Drop Hadoop 1 support*
>>>>> - We would no longer care about maintaining Hadoop 1 APIs (get rid
>>>>> of
>>>>> crappy reflection)
>>>>> - 2.2.0 (Hadoop 2 "stable") came out just under 1 year ago
>>>>> - Can be done for 1.7 or reconsidered for 2.0
>>>>>
>>>>>
>>>>>
>>>>> Do we already know if focusing on just Hadoop 2.2.0+ support will
>>>> result
>>>>
>>> in
>>>
>>>> any API impact?
>>>>
>>>>
>>>> I can't imagine it will impact our client API much, if any, but it'll
>>> certainly help simplify and fix bugs in our use of Hadoop (when they
>>> appear), and may result in fewer compatibility issues in MapReduce and
>>> elsewhere (dynamic class loading, reflection workarounds). It should also
>>> allow us to provide some guarantees in durability-related features
>>> (because
>>> we'll know they exist). It should also help simplify documentation and
>>> example configs, as well as reduce testing burdens.
>>>
>>> +1 to dropping Hadoop<2.2.0 in 1.7.0.
>>>
>>>
>>>
>> If it doesn't impact API or usage I'd also be +1. If it ends up leaking
>> out
>> somewhere, I'd want to take the opportunity of 1.7.0 to flag deprecations
>> so the transition can be made less severe (and remove in 2.0).
>>
>> Frankly, given the general expectations on durability we formed in 1.4 I'd
>> say it's dangerous for people to be running 1.5+ on Hadoop 1.
>>
>>
> Yeah, that's definitely what has to be weighed. Are we so aggressive to
> prevent users from even running on Hadoop 1 because it is inherently
> missing some of the sync mechanics needed to ensure no data loss regardless
> of redundant power?
>
> We warn users when they don't have the necessary options set at the HDFS
> level to properly sync, but we don't keep them from running like that.
>
> The more I think about it, I think I'd be more in favor of continuing to
> strongly encourage users to adopt Hadoop 2 and revisit dropping Hadoop 1
> for 2.0.0.
>
Re: 1.7 release timeline
Posted by Josh Elser <jo...@gmail.com>.
Sean Busbey wrote:
> On Wed, Oct 8, 2014 at 11:12 AM, Christopher<ct...@apache.org> wrote:
>
>> On Wed, Oct 8, 2014 at 9:47 AM, Sean Busbey<bu...@cloudera.com> wrote:
>>
>>> On Wed, Oct 8, 2014 at 12:13 AM, Josh Elser<jo...@gmail.com>
>> wrote:
>>>> Forgot one:
>>>>
>>>> *Drop Hadoop 1 support*
>>>> - We would no longer care about maintaining Hadoop 1 APIs (get rid of
>>>> crappy reflection)
>>>> - 2.2.0 (Hadoop 2 "stable") came out just under 1 year ago
>>>> - Can be done for 1.7 or reconsidered for 2.0
>>>>
>>>>
>>>>
>>> Do we already know if focusing on just Hadoop 2.2.0+ support will result
>> in
>>> any API impact?
>>>
>>>
>> I can't imagine it will impact our client API much, if any, but it'll
>> certainly help simplify and fix bugs in our use of Hadoop (when they
>> appear), and may result in fewer compatibility issues in MapReduce and
>> elsewhere (dynamic class loading, reflection workarounds). It should also
>> allow us to provide some guarantees in durability-related features (because
>> we'll know they exist). It should also help simplify documentation and
>> example configs, as well as reduce testing burdens.
>>
>> +1 to dropping Hadoop<2.2.0 in 1.7.0.
>>
>>
>
> If it doesn't impact API or usage I'd also be +1. If it ends up leaking out
> somewhere, I'd want to take the opportunity of 1.7.0 to flag deprecations
> so the transition can be made less severe (and remove in 2.0).
>
> Frankly, given the general expectations on durability we formed in 1.4 I'd
> say it's dangerous for people to be running 1.5+ on Hadoop 1.
>
Yeah, that's definitely what has to be weighed. Are we so aggressive to
prevent users from even running on Hadoop 1 because it is inherently
missing some of the sync mechanics needed to ensure no data loss
regardless of redundant power?
We warn users when they don't have the necessary options set at the HDFS
level to properly sync, but we don't keep them from running like that.
The more I think about it, I think I'd be more in favor of continuing to
strongly encourage users to adopt Hadoop 2 and revisit dropping Hadoop 1
for 2.0.0.
Re: 1.7 release timeline
Posted by Sean Busbey <bu...@cloudera.com>.
On Wed, Oct 8, 2014 at 11:12 AM, Christopher <ct...@apache.org> wrote:
> On Wed, Oct 8, 2014 at 9:47 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > On Wed, Oct 8, 2014 at 12:13 AM, Josh Elser <jo...@gmail.com>
> wrote:
> >
> > > Forgot one:
> > >
> > > *Drop Hadoop 1 support*
> > > - We would no longer care about maintaining Hadoop 1 APIs (get rid of
> > > crappy reflection)
> > > - 2.2.0 (Hadoop 2 "stable") came out just under 1 year ago
> > > - Can be done for 1.7 or reconsidered for 2.0
> > >
> > >
> > >
> > Do we already know if focusing on just Hadoop 2.2.0+ support will result
> in
> > any API impact?
> >
> >
> I can't imagine it will impact our client API much, if any, but it'll
> certainly help simplify and fix bugs in our use of Hadoop (when they
> appear), and may result in fewer compatibility issues in MapReduce and
> elsewhere (dynamic class loading, reflection workarounds). It should also
> allow us to provide some guarantees in durability-related features (because
> we'll know they exist). It should also help simplify documentation and
> example configs, as well as reduce testing burdens.
>
> +1 to dropping Hadoop <2.2.0 in 1.7.0.
>
>
If it doesn't impact API or usage I'd also be +1. If it ends up leaking out
somewhere, I'd want to take the opportunity of 1.7.0 to flag deprecations
so the transition can be made less severe (and remove in 2.0).
Frankly, given the general expectations on durability we formed in 1.4 I'd
say it's dangerous for people to be running 1.5+ on Hadoop 1.
--
Sean
Re: 1.7 release timeline
Posted by Christopher <ct...@apache.org>.
On Wed, Oct 8, 2014 at 9:47 AM, Sean Busbey <bu...@cloudera.com> wrote:
> On Wed, Oct 8, 2014 at 12:13 AM, Josh Elser <jo...@gmail.com> wrote:
>
> > Forgot one:
> >
> > *Drop Hadoop 1 support*
> > - We would no longer care about maintaining Hadoop 1 APIs (get rid of
> > crappy reflection)
> > - 2.2.0 (Hadoop 2 "stable") came out just under 1 year ago
> > - Can be done for 1.7 or reconsidered for 2.0
> >
> >
> >
> Do we already know if focusing on just Hadoop 2.2.0+ support will result in
> any API impact?
>
>
I can't imagine it will impact our client API much, if any, but it'll
certainly help simplify and fix bugs in our use of Hadoop (when they
appear), and may result in fewer compatibility issues in MapReduce and
elsewhere (dynamic class loading, reflection workarounds). It should also
allow us to provide some guarantees in durability-related features (because
we'll know they exist). It should also help simplify documentation and
example configs, as well as reduce testing burdens.
+1 to dropping Hadoop <2.2.0 in 1.7.0.
> > *Upgrade test script*
> >> - Keith mentioned there's some code from John McNamee that might help
> >> testing upgrade paths
> >>
> >>
> If we can make this a contrib, I have some code from Mike and I that we'd
> like to add.
>
+1
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
>
Re: 1.7 release timeline
Posted by Sean Busbey <bu...@cloudera.com>.
On Wed, Oct 8, 2014 at 12:13 AM, Josh Elser <jo...@gmail.com> wrote:
> Forgot one:
>
> *Drop Hadoop 1 support*
> - We would no longer care about maintaining Hadoop 1 APIs (get rid of
> crappy reflection)
> - 2.2.0 (Hadoop 2 "stable") came out just under 1 year ago
> - Can be done for 1.7 or reconsidered for 2.0
>
>
>
Do we already know if focusing on just Hadoop 2.2.0+ support will result in
any API impact?
> *Upgrade test script*
>> - Keith mentioned there's some code from John McNamee that might help
>> testing upgrade paths
>>
>>
If we can make this a contrib, I have some code from Mike and I that we'd
like to add.
--
Sean
Re: 1.7 release timeline
Posted by Josh Elser <jo...@gmail.com>.
Forgot one:
*Drop Hadoop 1 support*
- We would no longer care about maintaining Hadoop 1 APIs (get rid of
crappy reflection)
- 2.2.0 (Hadoop 2 "stable") came out just under 1 year ago
- Can be done for 1.7 or reconsidered for 2.0
Josh Elser wrote:
> Some more information on the subject. A few of us got together to
> co-work today and had an informal discussion on our individual interests
> for 1.7. Summary incoming:
>
> *Monitor re-write*
> - I was pushing this one, I think the monitor still has merit despite
> the goal of the desire of other to just integrate with external systems
> - I have some code in place, but still needs more work.
> - Is a unified/stable "metrics" API necessary for integration w/
> external tools? (or is JMX enough?)
> - An API would probably be a more usable interface than JMX
> - Such an API should be stateless (no log aggregation nor statistics
> over time)
> - Monitor still has uses for standalone/small deployments
> - If still being used, MVC approach would ease testing and addition of
> new data and views
> - Not necessary to hold up 1.7.0 from happening
>
> *Revisit performance*
> - Eric mentioned that he wants to spend some time running some Accumulo
> benchmarks, specifically YCSB.
> - Lots of related topics were mentioned that might be relevant
> * Other HDFS block cache implementations (HBase has lots of nice
> benchmarks, could learn from them)
> * A WIP patch for metadata updates have some promise (ACCUMULO-2889)
> * Collapse iterator stack (ACCUMULO-3079)
> * Possible improvements to Scanner for single-batch cases (reduce a few
> RPCs to one RPC)
> - Actual changes made likely to be found via investigation
> - Changing default conf values where relevant also mentioned
>
> *Distributed Tracing*
> - Billie has been spending some time working w/ some people on replacing
> Cloudtrace with HTrace
> - Mentioned that HTrace shares a remarkable amount of similarity with
> our existing tracing library
> - Upstream efforts in Hadoop-3 to integrate htrace to DN/NN calls
> - Some consideration given to replace traceserver with zipkin however
> not required for the first implementation
>
> *Decouple MiniAccumuloCluster from ITs*
> - Another one I've started working on
> - ITs are really great, we have a lot for really good cases
> - Running them against a real instance in infeasible right now
> - Would be good to express as many as possible in terms of only using
> Instance+Connector
> - Christopher mentioned possible benefit outside of tests to using the
> accumulo-maven-plugin as the "shim" between a real instance and a
> MiniAccumuloCluster
> - Some tests are written explicitly for MAC and must be ignored or run
> against a MAC when a real instance is available.
>
> *Upgrade test script*
> - Keith mentioned there's some code from John McNamee that might help
> testing upgrade paths
>
> *Hadoop Metrics2*
> - Metrics2 is the current library in use by Hadoop
> - Integration gives us a lot more flexibility, notably good integration
> with Ganglia provided (ACCUMULO-1817)
> - No one expressed interest in working on this directly (potential to slip)
>
> *Deprecate MockAccumulo?*
> - Talked about this for 1.6, decided against
> - It's now 1.7. Is it time?
> - Remember, deprecate != removal
>
> There are some outstanding things we need to investigate more:
> - Is improved JMX or metrics2 impl sufficient for integration with
> external monitoring tools? (considerations: nagios, ganglia, statsd,
> collectd, carbon, riemann... others?)
> - BatchWriter has some weird cases around error handling. Is intended
> that it survives failures, but that's very much not the case. Should
> probably be fixed around a major release, but need to figure out how
> exactly to fix it (needs someone to get behind it)
>
> If people want to continue discussion on these, let's break off
> individual topics into their own thread for clarity (and my sanity).
>
> Also, anyone have a desire to be "release manager"?
>
> - Josh
>
> Josh Elser wrote:
>> Thanks, John.
>>
>> I was thinking about trying to gun for January time-frame for a release.
>> I'd love to say before 2014 is over, but that probably just won't happen
>> for a major release with the holidays.
>>
>> For 1.7 right now, I see the following "bigger" items (correct me where
>> I'm wrong):
>>
>> * Replication (done)
>> * Upgrade rules/guarantees (proposed)
>> * Replace cloudtrace (in-progress)
>> * Rewrite monitor, include REST service (in-progress)
>> * Drop Hadoop 1 support (proposed)
>> * Decouple MiniAccumulo from ITs (in-progress)
>> * Other minicluster types: in-process, shim to real instance
>> (in-progress)
>> * Support Hadoop metrics2 (proposed)
>> * A few WAL/metadata related performance improvements (in-progress)
>>
>> Also, would be good to check the In-Progress state issues on JIRA. What
>> do people think?
>>
>> John Vines wrote:
>>> Moving this to it's own thread...
>>>
>>> On Mon, Oct 6, 2014 at 5:54 PM, Mike Drob<ma...@cloudera.com> wrote:
>>>
>>>> Related: Do we have a release timeline for 1.7?
>>>>
Re: 1.7 release timeline
Posted by Josh Elser <jo...@gmail.com>.
Some more information on the subject. A few of us got together to
co-work today and had an informal discussion on our individual interests
for 1.7. Summary incoming:
*Monitor re-write*
- I was pushing this one, I think the monitor still has merit despite
the goal of the desire of other to just integrate with external systems
- I have some code in place, but still needs more work.
- Is a unified/stable "metrics" API necessary for integration w/
external tools? (or is JMX enough?)
- An API would probably be a more usable interface than JMX
- Such an API should be stateless (no log aggregation nor statistics
over time)
- Monitor still has uses for standalone/small deployments
- If still being used, MVC approach would ease testing and addition
of new data and views
- Not necessary to hold up 1.7.0 from happening
*Revisit performance*
- Eric mentioned that he wants to spend some time running some
Accumulo benchmarks, specifically YCSB.
- Lots of related topics were mentioned that might be relevant
* Other HDFS block cache implementations (HBase has lots of nice
benchmarks, could learn from them)
* A WIP patch for metadata updates have some promise (ACCUMULO-2889)
* Collapse iterator stack (ACCUMULO-3079)
* Possible improvements to Scanner for single-batch cases (reduce a
few RPCs to one RPC)
- Actual changes made likely to be found via investigation
- Changing default conf values where relevant also mentioned
*Distributed Tracing*
- Billie has been spending some time working w/ some people on
replacing Cloudtrace with HTrace
- Mentioned that HTrace shares a remarkable amount of similarity with
our existing tracing library
- Upstream efforts in Hadoop-3 to integrate htrace to DN/NN calls
- Some consideration given to replace traceserver with zipkin however
not required for the first implementation
*Decouple MiniAccumuloCluster from ITs*
- Another one I've started working on
- ITs are really great, we have a lot for really good cases
- Running them against a real instance in infeasible right now
- Would be good to express as many as possible in terms of only using
Instance+Connector
- Christopher mentioned possible benefit outside of tests to using
the accumulo-maven-plugin as the "shim" between a real instance and a
MiniAccumuloCluster
- Some tests are written explicitly for MAC and must be ignored or
run against a MAC when a real instance is available.
*Upgrade test script*
- Keith mentioned there's some code from John McNamee that might help
testing upgrade paths
*Hadoop Metrics2*
- Metrics2 is the current library in use by Hadoop
- Integration gives us a lot more flexibility, notably good
integration with Ganglia provided (ACCUMULO-1817)
- No one expressed interest in working on this directly (potential to
slip)
*Deprecate MockAccumulo?*
- Talked about this for 1.6, decided against
- It's now 1.7. Is it time?
- Remember, deprecate != removal
There are some outstanding things we need to investigate more:
- Is improved JMX or metrics2 impl sufficient for integration with
external monitoring tools? (considerations: nagios, ganglia, statsd,
collectd, carbon, riemann... others?)
- BatchWriter has some weird cases around error handling. Is intended
that it survives failures, but that's very much not the case. Should
probably be fixed around a major release, but need to figure out how
exactly to fix it (needs someone to get behind it)
If people want to continue discussion on these, let's break off
individual topics into their own thread for clarity (and my sanity).
Also, anyone have a desire to be "release manager"?
- Josh
Josh Elser wrote:
> Thanks, John.
>
> I was thinking about trying to gun for January time-frame for a release.
> I'd love to say before 2014 is over, but that probably just won't happen
> for a major release with the holidays.
>
> For 1.7 right now, I see the following "bigger" items (correct me where
> I'm wrong):
>
> * Replication (done)
> * Upgrade rules/guarantees (proposed)
> * Replace cloudtrace (in-progress)
> * Rewrite monitor, include REST service (in-progress)
> * Drop Hadoop 1 support (proposed)
> * Decouple MiniAccumulo from ITs (in-progress)
> * Other minicluster types: in-process, shim to real instance (in-progress)
> * Support Hadoop metrics2 (proposed)
> * A few WAL/metadata related performance improvements (in-progress)
>
> Also, would be good to check the In-Progress state issues on JIRA. What
> do people think?
>
> John Vines wrote:
>> Moving this to it's own thread...
>>
>> On Mon, Oct 6, 2014 at 5:54 PM, Mike Drob<ma...@cloudera.com> wrote:
>>
>>> Related: Do we have a release timeline for 1.7?
>>>
Re: 1.7 release timeline
Posted by Josh Elser <jo...@gmail.com>.
Thanks, John.
I was thinking about trying to gun for January time-frame for a release.
I'd love to say before 2014 is over, but that probably just won't happen
for a major release with the holidays.
For 1.7 right now, I see the following "bigger" items (correct me where
I'm wrong):
* Replication (done)
* Upgrade rules/guarantees (proposed)
* Replace cloudtrace (in-progress)
* Rewrite monitor, include REST service (in-progress)
* Drop Hadoop 1 support (proposed)
* Decouple MiniAccumulo from ITs (in-progress)
* Other minicluster types: in-process, shim to real instance (in-progress)
* Support Hadoop metrics2 (proposed)
* A few WAL/metadata related performance improvements (in-progress)
Also, would be good to check the In-Progress state issues on JIRA. What
do people think?
John Vines wrote:
> Moving this to it's own thread...
>
> On Mon, Oct 6, 2014 at 5:54 PM, Mike Drob<ma...@cloudera.com> wrote:
>
>> Related: Do we have a release timeline for 1.7?
>>