[DISCUSS] Release 2.0.0

[Keith Turner <ke...@deenlo.com>]

I have been looking around at what may need to be done before a Fluo
release.  There were two issues w/ the 2.0.0 milestone, one was
already completed.  There other could not be done until after Accumulo
2.1.0 is released so I removed the 2.0.0 milestone from it.

I have been running the full build and fixed one reliably flaky IT.
There are some other flaky ones, but they are not as reliably flaky.

I successfully ran a small run of stresso on a single VM yesterday w/o
issue. I used Hadoop 3.1.3, ZK 3.4.14, and Accumulo 2.0.1.  I tried
using a newer version of ZK and ran into problems w/ Curator, ZK,
Hadoop and Accumulo.  These problems have nothing to do w/ Fluo, they
are just problems w/ the dependencies interacting w/ each other. Fluo
compiles fine against the old and new versions of ZK and Curator.  I
am not quite sure where it occurs, but at some point curator stopped
supporting ZK 3.4. I suspect these issues could be resolved at runtime
w/o any changes to Fluo. The following are the issues I recall
encountering.

 * Seems Mini Accumulo 2.0.0 does not support newer than ZK 3.4?  It
kept hanging w/ newer versions.
 * Newer versions of curator were trying to use a non-existant ZK
method.  Not exactly sure what was going on, some change between ZK
and Curator results in runtime exceptions.
 * I think newer version of Hadoop (maybe 3.3) are pulling in new
version of curator onto the classpath, which caused issues w/ older ZK
versions.  Not completely sure about this assumption, I may be wrong.
This problem was specific to stresso, which uses the yarn jar command
to run things.
 * Unrelated to curator, the Hadoop 3.1.4 shaded jars in maven seemed
to be empty.

I think the worst case of doing a 2.0.0 release now is that a 2.0.1
release w/ no API changes could possibly be needed to address possible
transitive dependency issues.

I think it would be nice to get the webindex example working against
2.0 before release, but I don't think it's required.  It's not using
any new APIs, so it would not test any new functionality.  I started
working on getting it in shape in the past and this required getting
Fluo Recipes up to date w/ the latest Accumulo 2.0 changes which I
did.  I also started getting Webindex up to date w/ Accumulo 2.0
changes, but did not complete that task.

There is always more work to do, but a lot has already been done for
2.0.0 and I think what is there is in really good shape and long
overdue for a release.

Re: [DISCUSS] Release 2.0.0

[Keith Turner <ke...@deenlo.com>]

On Tue, Jan 12, 2021 at 11:30 PM Christopher <ct...@apache.org> wrote:
>
> On Tue, Jan 12, 2021 at 10:26 AM Keith Turner <ke...@deenlo.com> wrote:
>
> > I have been looking around at what may need to be done before a Fluo
> > release.  There were two issues w/ the 2.0.0 milestone, one was
> > already completed.  There other could not be done until after Accumulo
> > 2.1.0 is released so I removed the 2.0.0 milestone from it.
> >
> > I have been running the full build and fixed one reliably flaky IT.
> > There are some other flaky ones, but they are not as reliably flaky.
> >
> > I successfully ran a small run of stresso on a single VM yesterday w/o
> > issue. I used Hadoop 3.1.3, ZK 3.4.14, and Accumulo 2.0.1.  I tried
> > using a newer version of ZK and ran into problems w/ Curator, ZK,
> > Hadoop and Accumulo.  These problems have nothing to do w/ Fluo, they
> > are just problems w/ the dependencies interacting w/ each other. Fluo
> > compiles fine against the old and new versions of ZK and Curator.  I
> > am not quite sure where it occurs, but at some point curator stopped
> > supporting ZK 3.4. I suspect these issues could be resolved at runtime
> > w/o any changes to Fluo. The following are the issues I recall
> > encountering.
> >
> >  * Seems Mini Accumulo 2.0.0 does not support newer than ZK 3.4?  It
> > kept hanging w/ newer versions.
> >
>
> I believe the main problem with minicluster was fixed in
> https://github.com/apache/accumulo/pull/1531, because minicluster requires
> the use of some four-letter word commands. This fix did not make it into
> 2.0.
>
>
> >  * Newer versions of curator were trying to use a non-existant ZK
> > method.  Not exactly sure what was going on, some change between ZK
> > and Curator results in runtime exceptions.
> >  * I think newer version of Hadoop (maybe 3.3) are pulling in new
> > version of curator onto the classpath, which caused issues w/ older ZK
> > versions.  Not completely sure about this assumption, I may be wrong.
> > This problem was specific to stresso, which uses the yarn jar command
> > to run things.
> >  * Unrelated to curator, the Hadoop 3.1.4 shaded jars in maven seemed
> > to be empty.
> >
> > I think the worst case of doing a 2.0.0 release now is that a 2.0.1
> > release w/ no API changes could possibly be needed to address possible
> > transitive dependency issues.
> >
> > I think it would be nice to get the webindex example working against
> > 2.0 before release, but I don't think it's required.  It's not using
> > any new APIs, so it would not test any new functionality.  I started
> > working on getting it in shape in the past and this required getting
> > Fluo Recipes up to date w/ the latest Accumulo 2.0 changes which I
> > did.  I also started getting Webindex up to date w/ Accumulo 2.0
> > changes, but did not complete that task.
> >
> > There is always more work to do, but a lot has already been done for
> > 2.0.0 and I think what is there is in really good shape and long
> > overdue for a release.
> >
>
> I'm inclined to agree. I think it would be fine to release with what is
> there now, as long as we're clear about what versions we've tried. I think
> you're right that most issues with newer versions could be addressed
> downstream with class path decisions. And, if there is something that can't
> be easily fixed, then releasing a 2.0.1 later would be reasonable.

I will look into spinning a RC

>
> If we're headed for a release, we should make sure the copyright dates are
> up-to-date, since it's now 2021.

Re: [DISCUSS] Release 2.0.0

[Christopher <ct...@apache.org>]

On Tue, Jan 12, 2021 at 10:26 AM Keith Turner <ke...@deenlo.com> wrote:

> I have been looking around at what may need to be done before a Fluo
> release.  There were two issues w/ the 2.0.0 milestone, one was
> already completed.  There other could not be done until after Accumulo
> 2.1.0 is released so I removed the 2.0.0 milestone from it.
>
> I have been running the full build and fixed one reliably flaky IT.
> There are some other flaky ones, but they are not as reliably flaky.
>
> I successfully ran a small run of stresso on a single VM yesterday w/o
> issue. I used Hadoop 3.1.3, ZK 3.4.14, and Accumulo 2.0.1.  I tried
> using a newer version of ZK and ran into problems w/ Curator, ZK,
> Hadoop and Accumulo.  These problems have nothing to do w/ Fluo, they
> are just problems w/ the dependencies interacting w/ each other. Fluo
> compiles fine against the old and new versions of ZK and Curator.  I
> am not quite sure where it occurs, but at some point curator stopped
> supporting ZK 3.4. I suspect these issues could be resolved at runtime
> w/o any changes to Fluo. The following are the issues I recall
> encountering.
>
>  * Seems Mini Accumulo 2.0.0 does not support newer than ZK 3.4?  It
> kept hanging w/ newer versions.
>

I believe the main problem with minicluster was fixed in
https://github.com/apache/accumulo/pull/1531, because minicluster requires
the use of some four-letter word commands. This fix did not make it into
2.0.


>  * Newer versions of curator were trying to use a non-existant ZK
> method.  Not exactly sure what was going on, some change between ZK
> and Curator results in runtime exceptions.
>  * I think newer version of Hadoop (maybe 3.3) are pulling in new
> version of curator onto the classpath, which caused issues w/ older ZK
> versions.  Not completely sure about this assumption, I may be wrong.
> This problem was specific to stresso, which uses the yarn jar command
> to run things.
>  * Unrelated to curator, the Hadoop 3.1.4 shaded jars in maven seemed
> to be empty.
>
> I think the worst case of doing a 2.0.0 release now is that a 2.0.1
> release w/ no API changes could possibly be needed to address possible
> transitive dependency issues.
>
> I think it would be nice to get the webindex example working against
> 2.0 before release, but I don't think it's required.  It's not using
> any new APIs, so it would not test any new functionality.  I started
> working on getting it in shape in the past and this required getting
> Fluo Recipes up to date w/ the latest Accumulo 2.0 changes which I
> did.  I also started getting Webindex up to date w/ Accumulo 2.0
> changes, but did not complete that task.
>
> There is always more work to do, but a lot has already been done for
> 2.0.0 and I think what is there is in really good shape and long
> overdue for a release.
>

I'm inclined to agree. I think it would be fine to release with what is
there now, as long as we're clear about what versions we've tried. I think
you're right that most issues with newer versions could be addressed
downstream with class path decisions. And, if there is something that can't
be easily fixed, then releasing a 2.0.1 later would be reasonable.

If we're headed for a release, we should make sure the copyright dates are
up-to-date, since it's now 2021.