Re: 2.1 Release TODO

[Mike Miller <mm...@apache.org>]

2.1 update for July and August.
7 PRs in progress.
1,253 marked as DONE. (+87)
6 left marked as a Blocker for 2.1. [1]
2 left as TODO for Compaction follow on work. [2]
1 left as TODO for ZK follow on work. [3]
Once #2197 is completed and merged, there will be some follow on work.

[1]:
https://github.com/apache/accumulo/issues?q=is%3Aissue%20is%3Aopen%20label%3Ablocker%20project%3Aapache%2Faccumulo%2F3
[2]: https://github.com/apache/accumulo/projects/13
[3]: https://github.com/apache/accumulo/projects/24

On Wed, Jun 29, 2022 at 9:11 AM Mike Miller <mm...@apache.org> wrote:

> It has been a month so I am sending a 2.1 update.
> 10 PRs in progress.
> 1,166 marked as DONE. (+36)
> 33 left in the TODO state. I went through these and dropped some from 2.1.
> https://github.com/apache/accumulo/projects/3
>
> 3 marked as TODO as Compaction follow on work:
> https://github.com/apache/accumulo/projects/13
>
> 2 in progress & 2 TODO for the Single Node ZK follow on work.
> https://github.com/apache/accumulo/projects/24
>
>
> On Fri, May 27, 2022 at 11:30 AM Mike Miller <mm...@apache.org> wrote:
>
>> Update on 2.1 progress. Come on Folks, let's Hold The Line. [1]
>> 14 Pull requests in progress. (-7 digression from last week)
>> 1,130 marked as DONE. (+19)
>> https://github.com/apache/accumulo/projects/3
>>
>> 3 marked as TODO as Compaction follow on work:
>> https://github.com/apache/accumulo/projects/13
>>
>> 2 in progress & 2 TODO for the Single Node ZK follow on work.
>> https://github.com/apache/accumulo/projects/24
>>
>> [1]: https://www.youtube.com/watch?v=htgr3pvBr-I
>>
>> On Fri, May 20, 2022 at 11:25 AM Mike Miller <mm...@apache.org> wrote:
>>
>>> Update on 2.1 progress.
>>> 7 Pull requests in progress.
>>> 1111 marked as DONE. ( I just missed sending this at 11:11 EST)
>>> https://github.com/apache/accumulo/projects/3
>>>
>>> 3 marked as TODO as Compaction follow on work:
>>> https://github.com/apache/accumulo/projects/13
>>>
>>> 2 marked as TODO for the Single Node ZK follow on work.
>>> https://github.com/apache/accumulo/projects/24
>>>
>>>
>>>
>>> On Fri, May 13, 2022 at 9:38 AM Mike Miller <mm...@apache.org> wrote:
>>>
>>>> Update on 2.1 progress.
>>>> 10 Pull Requests in progress.
>>>> 1,097 marked as DONE.
>>>> https://github.com/apache/accumulo/projects/3
>>>>
>>>> 3 Tickets marked as TODO as Compaction follow on work:
>>>> https://github.com/apache/accumulo/projects/13
>>>>
>>>> Only 1 Ticket marked as TODO for the ZK follow on work. I thought there
>>>> would be more here:
>>>> https://github.com/apache/accumulo/projects/24
>>>>
>>>> On Fri, Apr 8, 2022 at 7:00 AM Mike Miller <mm...@apache.org> wrote:
>>>>
>>>>> Update on 2.1 progress. For pull requests:
>>>>> 15 currently in progress.
>>>>> 32 are open as TODO. But a lot of these will get bumped to the next
>>>>> version.
>>>>> 1,025 DONE. Wow! Good work everyone.
>>>>>
>>>>> https://github.com/apache/accumulo/projects/3
>>>>>
>>>>> On Wed, Apr 6, 2022 at 4:55 PM Christopher <ct...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> After some additional consideration, and getting a better
>>>>>> understanding of
>>>>>> how the code is expected to work from discussing it with Dave... I'm a
>>>>>> little more inclined to support #2422 in 2.1, provided:
>>>>>>
>>>>>> 1. There's time for me to review it,
>>>>>> 2. It is sufficiently decoupled from the existing code and marked
>>>>>> experimental, so that we have the flexibility to alter its design, if
>>>>>> it
>>>>>> seems appropriate after it gets some exposure after the release,
>>>>>> 3. Unit tests and integration tests are reliably passing (as stable
>>>>>> as, or
>>>>>> more stable than, they are currently),
>>>>>> 4. No serious issues are discovered during review, and
>>>>>> 5. It doesn't delay a release past early June, as I think this is a
>>>>>> reasonable target date.
>>>>>>
>>>>>> This my wishlist before I can get behind it with a +1 for 2.1. If
>>>>>> these
>>>>>> aren't met, I do not intend to veto, but I'd be a -0 on its inclusion
>>>>>> to
>>>>>> 2.1. Of course, once I review it, my thoughts may change a bit.
>>>>>>
>>>>>> On Mon, Apr 4, 2022 at 7:07 PM Mike Miller <mm...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> > I think I can finish the FATE refactor PR [1] for 2.1. I had been
>>>>>> keeping
>>>>>> > it up to date with the latest in main but stopped because it was
>>>>>> too much
>>>>>> > work. I was waiting until the ZK property changes are completed
>>>>>> before
>>>>>> > resolving the latest conflicts. I don't think it is much of a risk.
>>>>>> It is
>>>>>> > mostly cleanup and refactoring to remove generics from the
>>>>>> serialization
>>>>>> > code. It will be some work to revisit but I think the risk is
>>>>>> pretty low.
>>>>>> > It would allow changing the serialization, which we may be able to
>>>>>> get into
>>>>>> > 2.1 as well.
>>>>>> >
>>>>>> > [1] https://github.com/apache/accumulo/pull/2475
>>>>>> >
>>>>>> > On Mon, Apr 4, 2022 at 11:50 AM Keith Turner <ke...@deenlo.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > > On Mon, Apr 4, 2022 at 11:17 AM Christopher <ct...@apache.org>
>>>>>> wrote:
>>>>>> > > >
>>>>>> > > > I haven't seen the metrics test fail very often lately. If it's
>>>>>> > stable, I
>>>>>> > > > don't mind removing the blocker on that issue, but I'd be
>>>>>> reluctant to
>>>>>> > > > close it entirely just yet, until we can verify it doesn't
>>>>>> happen
>>>>>> > > anymore.
>>>>>> > > >
>>>>>> > > > As for the original list of potential issues to include, I'm in
>>>>>> favor
>>>>>> > of
>>>>>> > > > trying to get #2197 in. It was started awhile ago, is
>>>>>> relatively simple
>>>>>> > > and
>>>>>> > > > well understood by several of us already... it just needs a bit
>>>>>> of
>>>>>> > > > attention to finalize reviews so it can be merged.
>>>>>> > > >
>>>>>> > > > However, I'm reluctant to include #2422, because I don't think
>>>>>> it's
>>>>>> > near
>>>>>> > > > ready enough, and by the time it is, it will be very last
>>>>>> minute, and I
>>>>>> > > > don't want to delay 2.1 further for it. Even if it's included
>>>>>> as an
>>>>>> > > > experimental feature, I think it has huge potential to be
>>>>>> disruptive,
>>>>>> > or
>>>>>> > > to
>>>>>> > > > have a lot of churn by the time people actually have a chance
>>>>>> to review
>>>>>> > > it
>>>>>> > > > thoroughly. Furthermore, I think there are possible
>>>>>> alternatives (like
>>>>>> > a
>>>>>> > > > fully client-side implementation, based on offline scanners)
>>>>>> that would
>>>>>> > > > avoid the tight coupling of a new service to Accumulo's core
>>>>>> code. This
>>>>>> > >
>>>>>> > > There are some advantages to scan servers over direct file access
>>>>>> to
>>>>>> > > consider.  One is scalability of computation, if a web server is
>>>>>> > > serving N client queries with scan servers those can potentially
>>>>>> go to
>>>>>> > > different scan servers.  With direct file access, all N queries
>>>>>> and
>>>>>> > > their iterator stacks would have to run in the web server.
>>>>>> Another is
>>>>>> > > scalability of caching/memory.  When web servers send queries to
>>>>>> scan
>>>>>> > > servers using a sticky algorithm for assigning tablets to groups
>>>>>> of
>>>>>> > > scan servers, it could lead to good cache utilization and sharing
>>>>>> that
>>>>>> > > may not be possible when running scans directly in the web
>>>>>> server. So
>>>>>> > > scan servers allow scaling cache and computations for queries
>>>>>> > > independently of web servers in way that may not be possible with
>>>>>> > > direct file access.
>>>>>> > >
>>>>>> > > Another advantage to consider is isolation.  With direct file
>>>>>> access
>>>>>> > > and queries running directly in a web server, a bad query could
>>>>>> bring
>>>>>> > > down a web server and lots of unrelated queries.  Having a bad
>>>>>> query
>>>>>> > > bring down a scan server may be less disruptive.
>>>>>> > >
>>>>>> > > > thread isn't for discussing this in depth, so we can have that
>>>>>> > discussion
>>>>>> > > > in a separate thread, but I'm generally opposed to including it
>>>>>> this
>>>>>> > late
>>>>>> > > > in 2.1's development, given the timing, size and scope, tight
>>>>>> coupling,
>>>>>> > > and
>>>>>> > > > current state.
>>>>>> > > >
>>>>>> > > > I don't know enough about #2475 to have a strong opinion, but
>>>>>> it looks
>>>>>> > > big,
>>>>>> > > > and possibly high-risk, given the critical code it touches. It
>>>>>> > currently
>>>>>> > > > has a substantial number of conflicts with the main branch.
>>>>>> However, I
>>>>>> > > was
>>>>>> > > > thinking that *some* minimal refactoring (like low-risk
>>>>>> automatic
>>>>>> > > > refactoring, like moving packages) could be done. So, if that's
>>>>>> all
>>>>>> > this
>>>>>> > > > does, it might be okay. Otherwise, maybe it can be simplified?
>>>>>> At the
>>>>>> > > very
>>>>>> > > > least, I was thinking it would be a good opportunity to move the
>>>>>> > > > `org.apache.accumulo.fate` packages into an appropriate
>>>>>> > > > `org.apache.accumulo.core` parent package (some would go to
>>>>>> > > o.a.a.core.fate
>>>>>> > > > and others might go to o.a.a.core.util or similar) to keep the
>>>>>> package
>>>>>> > > > namespaces standardized, which is helpful to avoid naming
>>>>>> collisions
>>>>>> > and
>>>>>> > > > jar sealing issues, as well as for less complicated jigsaw
>>>>>> module
>>>>>> > > > definitions in future. Since 2.1 FaTE is already incompatible
>>>>>> with
>>>>>> > prior
>>>>>> > > > versions, a rename at this time would be less disruptive.
>>>>>> > > >
>>>>>> > > > Another task I had wanted to be done for 2.1, before I got
>>>>>> distracted
>>>>>> > > > fixing test failures during and after Christmas and trying to
>>>>>> work
>>>>>> > > through
>>>>>> > > > the singleton manager zookeeper stuff to see what we could
>>>>>> simplify.
>>>>>> > > What I
>>>>>> > > > had wanted done was to standardize the way we pass table
>>>>>> identifiers
>>>>>> > > (name,
>>>>>> > > > IDs) across the RPC layer, since we currently do that
>>>>>> inconsistently. I
>>>>>> > > > don't remember if there's an existing ticket open for it, but I
>>>>>> have a
>>>>>> > > > working branch I had started working out of for it before
>>>>>> Christmas.
>>>>>> > It's
>>>>>> > > > relatively simple work, and would set us up for some much
>>>>>> better APIs
>>>>>> > > going
>>>>>> > > > forward, as well as help with logging information about table
>>>>>> actions.
>>>>>> > If
>>>>>> > > > necessary, it could be bumped to a future version, but then
>>>>>> we'd have
>>>>>> > > more
>>>>>> > > > churn in the thrift layer. So, I'd prefer to get it for 2.1 to
>>>>>> avoid
>>>>>> > > that.
>>>>>> > > >
>>>>>> > > > As for planning, I was thinking early May for a code freeze
>>>>>> (except bug
>>>>>> > > > fixes and small improvements found during testing), so we can
>>>>>> try to
>>>>>> > > > release towards the end of May/early June. If we go with that
>>>>>> timeline,
>>>>>> > > > that's not a lot of time to wrap up features and have time for
>>>>>> > > > review/testing, so we may need to be selective about what we
>>>>>> hold off
>>>>>> > > until
>>>>>> > > > the next version, unless we want to further delay 2.1.
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > On Mon, Apr 4, 2022 at 9:13 AM Dave Marion <dmarion18@gmail.com
>>>>>> >
>>>>>> > wrote:
>>>>>> > > >
>>>>>> > > > > I think [3] is OBE and can be closed.
>>>>>> > > > >
>>>>>> > > > > On Mon, Apr 4, 2022 at 9:11 AM Mike Miller <
>>>>>> mmiller@apache.org>
>>>>>> > wrote:
>>>>>> > > > >
>>>>>> > > > > > Yes I agree, that was the goal of this email thread. I
>>>>>> found a few
>>>>>> > > more
>>>>>> > > > > > tickets that should be addressed for the next release.
>>>>>> > > > > >
>>>>>> > > > > > Ivan - There was some work done on this PR but it has been
>>>>>> some
>>>>>> > > time. Do
>>>>>> > > > > > you want to take a look at it? Implement a Thread limit. [1]
>>>>>> > > > > > Keith T - I think we should get this one merged to fix that
>>>>>> > > consistency
>>>>>> > > > > > check bug I found. It looks like it is finished. [2]
>>>>>> > > > > > Dave & Dom - Were you guys able to figure out a fix for the
>>>>>> new
>>>>>> > > external
>>>>>> > > > > > compaction metrics test? [3]
>>>>>> > > > > >
>>>>>> > > > > > FYI we have 6 blockers for 2.1:
>>>>>> > > > > > https://github.com/apache/accumulo/labels/blocker
>>>>>> > > > > >
>>>>>> > > > > > This is almost definitely going into 2.1 [4]. Thanks Jeff!
>>>>>> > > > > >
>>>>>> > > > > > [1] https://github.com/apache/accumulo/pull/1487
>>>>>> > > > > > [2] https://github.com/apache/accumulo/pull/2574
>>>>>> > > > > > [3] https://github.com/apache/accumulo/issues/2406
>>>>>> > > > > > [4] https://github.com/apache/accumulo/pull/2215
>>>>>> > > > > >
>>>>>> > > > > > On Fri, Apr 1, 2022 at 2:21 PM Dave Marion <
>>>>>> dmarion18@gmail.com>
>>>>>> > > wrote:
>>>>>> > > > > >
>>>>>> > > > > > > I think it would be useful to do some release planning so
>>>>>> that we
>>>>>> > > know
>>>>>> > > > > > what
>>>>>> > > > > > > features we are working towards and in which release they
>>>>>> will be
>>>>>> > > in.
>>>>>> > > > > > This
>>>>>> > > > > > > would be helpful for determining what existing PRs need
>>>>>> to make
>>>>>> > it
>>>>>> > > into
>>>>>> > > > > > > 2.1.0. 2.1.0 is the LTM release, so patches for existing
>>>>>> features
>>>>>> > > will
>>>>>> > > > > be
>>>>>> > > > > > > backported (2.1.1, 2.1.2, 2.1.3, etc.) However, as
>>>>>> defined in
>>>>>> > [1],
>>>>>> > > > > > features
>>>>>> > > > > > > that don't make it into 2.1.0 will go into the next
>>>>>> non-LTM
>>>>>> > release
>>>>>> > > > > > (2.2.0)
>>>>>> > > > > > > and any patches to bugs in those features will go into
>>>>>> the next
>>>>>> > > non-LTM
>>>>>> > > > > > > release after that (2.3.0).
>>>>>> > > > > > >
>>>>>> > > > > > > I'm not trying to hold up the 2.1.0 release by suggesting
>>>>>> that we
>>>>>> > > > > perform
>>>>>> > > > > > > this activity. I'm just asking what the future holds,
>>>>>> even if
>>>>>> > it's
>>>>>> > > just
>>>>>> > > > > > one
>>>>>> > > > > > > feature in the next non-LTM release. My concern is that
>>>>>> the next
>>>>>> > > > > release
>>>>>> > > > > > > will be open-ended and anything not included in 2.1.0
>>>>>> might not
>>>>>> > > get put
>>>>>> > > > > > > into a release for a very long time.
>>>>>> > > > > > >
>>>>>> > > > > > > [1]
>>>>>> https://accumulo.apache.org/contributor/versioning.html#LTM
>>>>>> > > > > > >
>>>>>> > > > > > >
>>>>>> > > > > > > On Thu, Mar 31, 2022 at 11:43 AM Mike Miller <
>>>>>> mmiller@apache.org
>>>>>> > >
>>>>>> > > > > wrote:
>>>>>> > > > > > >
>>>>>> > > > > > > > Starting an email chain of things that folks want to
>>>>>> finish for
>>>>>> > > 2.1.
>>>>>> > > > > > Here
>>>>>> > > > > > > > is what we currently have in the works that are most
>>>>>> likely
>>>>>> > going
>>>>>> > > > > into
>>>>>> > > > > > > 2.1:
>>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2569
>>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2600
>>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2293
>>>>>> > > > > > > >
>>>>>> > > > > > > > Some things that may go into 2.1:
>>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2422
>>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2475
>>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2197
>>>>>> > > > > > > >
>>>>>> > > > > > > > I created a Project for follow on work to the ZK
>>>>>> property
>>>>>> > > change. I
>>>>>> > > > > was
>>>>>> > > > > > > > planning on putting tasks in there that we want to
>>>>>> complete for
>>>>>> > > 2.1.
>>>>>> > > > > > But
>>>>>> > > > > > > we
>>>>>> > > > > > > > could also use it for post 2.1 work.
>>>>>> > > > > > > > https://github.com/apache/accumulo/projects/24
>>>>>> > > > > > > > https://github.com/apache/accumulo/issues/2469
>>>>>> > > > > > > >
>>>>>> > > > > > > > FYI a draft copy of the release notes has already been
>>>>>> on the
>>>>>> > > > > website:
>>>>>> > > > > > > > https://accumulo.apache.org/release/accumulo-2.1.0/
>>>>>> > > > > > > >
>>>>>> > > > > > > > This may be a good thread to discuss whether or not a
>>>>>> task
>>>>>> > needs
>>>>>> > > to
>>>>>> > > > > go
>>>>>> > > > > > > into
>>>>>> > > > > > > > 2.1 or should wait for the next version. We currently
>>>>>> have 32
>>>>>> > > open
>>>>>> > > > > pull
>>>>>> > > > > > > > requests so please email me if there is one that you
>>>>>> would like
>>>>>> > > > > > > prioritized
>>>>>> > > > > > > > for 2.1.
>>>>>> > > > > > > >
>>>>>> > > > > > >
>>>>>> > > > > >
>>>>>> > > > >
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>