Re: [DISCUSS] For 3.0.0-beta-2 and the final 3.0.0 release

[Istvan Toth <st...@cloudera.com.INVALID>]

Sorry, I missed your reply eralier, Duo.

I still haven't figured out the issue (though at least now I am now able to
run those tests from a debugger)
No ticket yet, as I'm still not sure which project to use.

To repro this, we need to
* install a current HBase 3 snapshot locally from branch-3 head,
* check out my WIP Phoenix HBase 3 branch:
https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025
* then run a faling test:

*mvn clean verify -am -pl phoenix-core
-Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray
-Dhbase.profile=3.0  -Dtest=foo*
There are other similar tests failures, missing expected ResultSet rows
after aggressively moving around the table regions during the query.

Istvan

On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> Is there an issue in HBase or Phoenix for tracking this problem?
>
> Thanks.
>
> Istvan Toth <st...@cloudera.com.invalid> 于2025年10月16日周四 11:58写道：
> >
> > Quick heads up:
> > I'm investigating a regression in the Phoenix test suite where scans are
> > not reset when the processed region is moved, leading to missing data
> from
> > the scan.
> >
> > I am not positive yet that the issue is on the HBase side, will report
> here
> > (and open a ticket if needed) when I know more.
> >
> > Istvan
> >
> > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
> >
> > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week to
> > > see if we can get these issues done, otherwise will move them out of
> > > 3.0.0.
> > >
> > > And the HBCK2 issue has been fixed.
> > >
> > > Plan to cut a new hbase-operator-tools release before cutting
> 3.0.0-beta-2.
> > >
> > > Thanks.
> > >
> > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年8月31日周日 22:57写道：
> > > >
> > > > I've finished one round of reviewing pending issues for 3.0.0-beta-2.
> > > >
> > > > There are still some necessary ones we need to get in before the
> final
> > > > 3.0.0 release, like make HBCK2 work against hbase3.
> > > >
> > > > Will keep working on them.
> > > >
> > > > Thanks.
> > > >
> > > > Nihal Jain <ni...@gmail.com> 于2025年6月19日周四 16:06写道：
> > > > >
> > > > > Yes, from hbase-thirdparty side, release is done and also consumed
> in
> > > hbase
> > > > > branch-2.5 and branch-2.6 and we are good here.
> > > > >
> > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) <
> palomino219@gmail.com>
> > > wrote:
> > > > >
> > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have
> already
> > > > > > made the new thirdparty release without jetty 12.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Nihal Jain <ni...@gmail.com> 于2025年6月18日周三 01:51写道：
> > > > > > >
> > > > > > > Ah yes, let me restart work for hbase thirdparty release for
> jetty
> > > 12
> > > > > > soon.
> > > > > > >
> > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth
> > > <stoty@cloudera.com.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I think we should make sure we're using Jetty 12 in the next
> > > beta. (
> > > > > > so we
> > > > > > > > need a thirdparty release for Jetty 12)
> > > > > > > > We could also review other dependencies, and bump the ones
> (if
> > > any)
> > > > > > that
> > > > > > > > were kept on old versions because of the Java 8 requirement.
> > > > > > > >
> > > > > > > > Istvan
> > > > > > > >
> > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) <
> > > palomino219@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > The good news is that I've finally got a successful 10B
> ITBLL
> > > run.
> > > > > > > > >
> > > > > > > > > You can check HBASE-27919 for more details.
> > > > > > > > >
> > > > > > > > > Next let me cleanup the jira issues and nightly runs.
> > > > > > > > >
> > > > > > > > > This time I think we can see 3.0.0 out soon.
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > >
> > > > > > > > > Nihal Jain <ni...@apache.org> 于2025年3月12日周三 03:37写道：
> > > > > > > > > >
> > > > > > > > > > Hi Duo,
> > > > > > > > > >
> > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I
> encountered
> > > > > > HBASE-29144
> > > > > > > > > with kerberos. Posting here to just bring to attention.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Nihal
> > > > > > > > > >
> > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote:
> > > > > > > > > > > A good news, cluster replication worked fine while
> rolling
> > > > > > upgrading
> > > > > > > > > in my test.
> > > > > > > > > > >
> > > > > > > > > > > Let me focus on the region replication stuff. After
> fixing
> > > this
> > > > > > > > issue,
> > > > > > > > > > > the only blocker is a stable 10B ITBLL run.
> > > > > > > > > > >
> > > > > > > > > > > Hope we could get 3.0.0 out soon.
> > > > > > > > > > >
> > > > > > > > > > > Thanks.
> > > > > > > > > > >
> > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年3月6日周四
> > > 22:55写道：
> > > > > > > > > > > >
> > > > > > > > > > > > I've finished one round of rolling upgrading test. It
> > > worked
> > > > > > fine.
> > > > > > > > > > > > Please see HBASE-29161.
> > > > > > > > > > > >
> > > > > > > > > > > > But soon I found a problem when memstore replication
> is
> > > > > > enabled,
> > > > > > > > and
> > > > > > > > > > > > filed HBASE-29169.
> > > > > > > > > > > >
> > > > > > > > > > > > I will test upgrading with cluster replication too,
> to
> > > see if
> > > > > > there
> > > > > > > > > > > > are any other issues.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > >
> > > > > > > > > > > > Istvan Toth <st...@apache.org> 于2025年1月20日周一
> 14:24写道：
> > > > > > > > > > > > >
> > > > > > > > > > > > > For HBASE-28431, only the final step of adding the
> > > > > > Hadoop-less
> > > > > > > > > assembly is
> > > > > > > > > > > > > missing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please review
> > > https://github.com/apache/hbase/pull/5766 if
> > > > > > you
> > > > > > > > > have the
> > > > > > > > > > > > > time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Istvan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) <
> > > > > > > > > palomino219@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I've finished the first round of issue
> alignment, it
> > > took
> > > > > > > > > several months...
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > where we can have several ignore files, so we do
> not
> > > need
> > > > > > to
> > > > > > > > > manually
> > > > > > > > > > > > > > check the result every time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So now, there are still several things to be done
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Make sure we have all the features and fixes
> we
> > > want to
> > > > > > land
> > > > > > > > > on
> > > > > > > > > > > > > > branch-3. For example, I think Istvan's assembly
> > > cleanup
> > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > > > included. Please report other issues you want to
> > > land in
> > > > > > final
> > > > > > > > > 3.0.0.
> > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to
> see
> > > if I
> > > > > > can
> > > > > > > > get
> > > > > > > > > > > > > > enough resources but it will be great if others
> > > could help
> > > > > > > > here.
> > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling
> > > upgrade
> > > > > > test
> > > > > > > > > with the
> > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have
> big
> > > changes
> > > > > > on
> > > > > > > > > > > > > > replication and region replica replication, I
> plan
> > > to have
> > > > > > > > tests
> > > > > > > > > with
> > > > > > > > > > > > > > these two features enabled too if possible.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Any other suggestions?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> 于2024年9月19日周四
> > > > > > 22:35写道：
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > And do we want to apply the same trick to
> > > MasterObserver
> > > > > > and
> > > > > > > > > > > > > > > RegionServerObserver?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For RegionObserver, we'd better do this because
> > > there are
> > > > > > > > > already
> > > > > > > > > > > > > > > methods in it that use the ObserverContext<?
> > > extends
> > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration,
> which
> > > were
> > > > > > added
> > > > > > > > in
> > > > > > > > > > > > > > > HBASE-10378.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For other coprocessor observers, we do not have
> > > such
> > > > > > methods
> > > > > > > > > yet.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> 于2024年9月19日周四
> > > > > > 22:03写道：
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm
> inclined
> > > toward
> > > > > > doing
> > > > > > > > > it here
> > > > > > > > > > > > > > > > because (i think) we expect coprocessor
> > > > > > implementations to
> > > > > > > > > necessarily
> > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not
> thought
> > > > > > through
> > > > > > > > > what
> > > > > > > > > > > > > > features are
> > > > > > > > > > > > > > > > enabled by this new generic signature.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo
> Zhang) <
> > > > > > > > > palomino219@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Filed HBASE-28862.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It will break old coprocessor
> implementations
> > > even if
> > > > > > > > they
> > > > > > > > > do not use
> > > > > > > > > > > > > > > > > any deprecated method. But since it does
> not
> > > change
> > > > > > any
> > > > > > > > > abilities, I
> > > > > > > > > > > > > > > > > think it is OK to include this in a major
> > > release as
> > > > > > we
> > > > > > > > > have already
> > > > > > > > > > > > > > > > > done bunch of breaking changes, like
> changing
> > > the
> > > > > > > > protobuf
> > > > > > > > > from
> > > > > > > > > > > > > > > > > unshaded one to shaded and relocated one...
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> > > 于2024年9月19日周四
> > > > > > > > > 17:36写道：
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me I
> > > think we
> > > > > > > > > should do this
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > 3.0.0.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> > > 于2024年9月19日周四
> > > > > > > > > 16:46写道：
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Should we take the opportunity to
> upgrade
> > > the
> > > > > > generic
> > > > > > > > > > > > > > definitions in
> > > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > > coprocessor signatures as well?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > >
> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo
> Zhang <
> > > > > > > > > zhangduo@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I've done several rounds of API
> cleanup
> > > in
> > > > > > > > > HBASE-24888 and now
> > > > > > > > > > > > > > for me
> > > > > > > > > > > > > > > > > > > > I think it is enough for making the
> new
> > > major
> > > > > > > > > release.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning
> the
> > > commit
> > > > > > > > > history and jira
> > > > > > > > > > > > > > > > > > > > issues, it will be a huge project
> for a
> > > major
> > > > > > > > > release and may
> > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > several weeks.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > After that, I think we are good to
> go.
> > > And I
> > > > > > will
> > > > > > > > > also try to
> > > > > > > > > > > > > > find
> > > > > > > > > > > > > > > > > > > > some resources to run several rounds
> of
> > > ITBLL
> > > > > > and
> > > > > > > > > also some
> > > > > > > > > > > > > > upgrading
> > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is
> OK,
> > > I will
> > > > > > > > > release
> > > > > > > > > > > > > > 3.0.0-beta-2,
> > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the
> > > 3.0.0-beta-2
> > > > > > tag
> > > > > > > > > and make the
> > > > > > > > > > > > > > 3.0.0
> > > > > > > > > > > > > > > > > > > > release soon.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hope I can get all these things done
> > > within the
> > > > > > > > year
> > > > > > > > > 2024.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > > > > *Email*: stoty@cloudera.com
> > > > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > > [image:
> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > > [image:
> > > > > > Cloudera
> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > > > > ------------------------------
> > > > > > > > ------------------------------
> > > > > > > >
> > > > > >
> > >
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------
>


-- 
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------

Re: [DISCUSS] For 3.0.0-beta-2 and the final 3.0.0 release

["张铎(Duo Zhang)" <pa...@gmail.com>]

Going to cut 3.0.0-beta-2 right now.

Moved some issues out of 3.0.0-beta-2.

Thanks.

张铎(Duo Zhang) <pa...@gmail.com> 于2026年6月6日周六 17:12写道：
>
> The JUnit5 migration is nearly done.
>
> I will start new rounds of 10B ITBLL to verify the stability of branch-3.
>
> After merging the last several PRs for migration and the ITBLL result
> is good, I will start cutting the 3.0.0-beta-2 release.
>
> Thanks.
>
> 张铎(Duo Zhang) <pa...@gmail.com> 于2026年3月30日周一 19:28写道：
> >
> > Some updates here, there is no blocker issue for 3.0.0-beta-2 release
> > now, but the junit5 upgrading is in its last stage, where we want to
> > migrate all the tests in hbase-server, so I plan to finish the
> > migration before cutting the final release.
> >
> > I will also start to run several rounds of 10B ITBLL to check the
> > stability of branch-3.
> >
> > Thanks.
> >
> > 张铎(Duo Zhang) <pa...@gmail.com> 于2026年1月22日周四 19:15写道：
> > >
> > > Thanks Istvan for the detailed analysis!
> > >
> > > For HBASE-29834, I've already put up a PR to fix the problem. See
> > >
> > > https://github.com/apache/hbase/pull/7660
> > >
> > > And on HBASE-29843, I do not fully understand the problem and need
> > > more context about the usage on Phoenix side. I think we should be
> > > able to fix it.
> > >
> > > Thanks.
> > >
> > > Istvan Toth <st...@cloudera.com> 于2026年1月22日周四 14:33写道：
> > > >
> > > > This seems to have been caused by forward synchronous RPC changes between
> > > > Hbase 2 and 3 (where HBase 2 simulates non-inclusive start keys by
> > > > adding a \00 byte and setting them as inclusive, while HBase 3 handles them
> > > > natively.
> > > > This is something that we can handle on the Phoenix side.
> > > >
> > > > On the other hand, I've found a major and a minor breaking change in HBase
> > > > 3:
> > > >
> > > > The big problem is HBASE-29843
> > > > <https://issues.apache.org/jira/browse/HBASE-29843>
> > > > HBaseRPCController.setPriority(final TableName tn) is never called on
> > > > branch-3+
> > > > Phoenix relies on setting it its own HBaseRPCController to set custom
> > > > priorities, but HBase 3 has mostly abandoned using HBaseRPCController,
> > > > leaving no
> > > > working mechanism for Phoenix to use.
> > > > HBase needs to restore using it to maintain compatibility. (or provide some
> > > > alternative method)
> > > >
> > > > The smaller problem is HBASE-29834
> > > > <https://issues.apache.org/jira/browse/HBASE-29834> MILLIS_BETWEEN_NEXTS
> > > > metric is not updated on branch-3+
> > > > This does not break Phoenix, it just happens to be detected by a Phoenix
> > > > test.
> > > >
> > > > Istvan
> > > >
> > > >
> > > > On Mon, Nov 3, 2025 at 10:22 AM Istvan Toth <st...@cloudera.com> wrote:
> > > >
> > > > > Sorry, I missed your reply eralier, Duo.
> > > > >
> > > > > I still haven't figured out the issue (though at least now I am now able
> > > > > to run those tests from a debugger)
> > > > > No ticket yet, as I'm still not sure which project to use.
> > > > >
> > > > > To repro this, we need to
> > > > > * install a current HBase 3 snapshot locally from branch-3 head,
> > > > > * check out my WIP Phoenix HBase 3 branch:
> > > > > https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025
> > > > > * then run a faling test:
> > > > >
> > > > > *mvn clean verify -am -pl phoenix-core
> > > > > -Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray
> > > > > -Dhbase.profile=3.0  -Dtest=foo*
> > > > > There are other similar tests failures, missing expected ResultSet rows
> > > > > after aggressively moving around the table regions during the query.
> > > > >
> > > > > Istvan
> > > > >
> > > > > On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Is there an issue in HBase or Phoenix for tracking this problem?
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >> Istvan Toth <st...@cloudera.com.invalid> 于2025年10月16日周四 11:58写道：
> > > > >> >
> > > > >> > Quick heads up:
> > > > >> > I'm investigating a regression in the Phoenix test suite where scans are
> > > > >> > not reset when the processed region is moved, leading to missing data
> > > > >> from
> > > > >> > the scan.
> > > > >> >
> > > > >> > I am not positive yet that the issue is on the HBase side, will report
> > > > >> here
> > > > >> > (and open a ticket if needed) when I know more.
> > > > >> >
> > > > >> > Istvan
> > > > >> >
> > > > >> > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week to
> > > > >> > > see if we can get these issues done, otherwise will move them out of
> > > > >> > > 3.0.0.
> > > > >> > >
> > > > >> > > And the HBCK2 issue has been fixed.
> > > > >> > >
> > > > >> > > Plan to cut a new hbase-operator-tools release before cutting
> > > > >> 3.0.0-beta-2.
> > > > >> > >
> > > > >> > > Thanks.
> > > > >> > >
> > > > >> > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年8月31日周日 22:57写道：
> > > > >> > > >
> > > > >> > > > I've finished one round of reviewing pending issues for
> > > > >> 3.0.0-beta-2.
> > > > >> > > >
> > > > >> > > > There are still some necessary ones we need to get in before the
> > > > >> final
> > > > >> > > > 3.0.0 release, like make HBCK2 work against hbase3.
> > > > >> > > >
> > > > >> > > > Will keep working on them.
> > > > >> > > >
> > > > >> > > > Thanks.
> > > > >> > > >
> > > > >> > > > Nihal Jain <ni...@gmail.com> 于2025年6月19日周四 16:06写道：
> > > > >> > > > >
> > > > >> > > > > Yes, from hbase-thirdparty side, release is done and also
> > > > >> consumed in
> > > > >> > > hbase
> > > > >> > > > > branch-2.5 and branch-2.6 and we are good here.
> > > > >> > > > >
> > > > >> > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) <
> > > > >> palomino219@gmail.com>
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have
> > > > >> already
> > > > >> > > > > > made the new thirdparty release without jetty 12.
> > > > >> > > > > >
> > > > >> > > > > > Thanks.
> > > > >> > > > > >
> > > > >> > > > > > Nihal Jain <ni...@gmail.com> 于2025年6月18日周三 01:51写道：
> > > > >> > > > > > >
> > > > >> > > > > > > Ah yes, let me restart work for hbase thirdparty release for
> > > > >> jetty
> > > > >> > > 12
> > > > >> > > > > > soon.
> > > > >> > > > > > >
> > > > >> > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth
> > > > >> > > <stoty@cloudera.com.invalid
> > > > >> > > > > > >
> > > > >> > > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > I think we should make sure we're using Jetty 12 in the next
> > > > >> > > beta. (
> > > > >> > > > > > so we
> > > > >> > > > > > > > need a thirdparty release for Jetty 12)
> > > > >> > > > > > > > We could also review other dependencies, and bump the ones
> > > > >> (if
> > > > >> > > any)
> > > > >> > > > > > that
> > > > >> > > > > > > > were kept on old versions because of the Java 8 requirement.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Istvan
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) <
> > > > >> > > palomino219@gmail.com>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > The good news is that I've finally got a successful 10B
> > > > >> ITBLL
> > > > >> > > run.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > You can check HBASE-27919 for more details.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Next let me cleanup the jira issues and nightly runs.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > This time I think we can see 3.0.0 out soon.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Nihal Jain <ni...@apache.org> 于2025年3月12日周三 03:37写道：
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Hi Duo,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I
> > > > >> encountered
> > > > >> > > > > > HBASE-29144
> > > > >> > > > > > > > > with kerberos. Posting here to just bring to attention.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Regards
> > > > >> > > > > > > > > > Nihal
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote:
> > > > >> > > > > > > > > > > A good news, cluster replication worked fine while
> > > > >> rolling
> > > > >> > > > > > upgrading
> > > > >> > > > > > > > > in my test.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Let me focus on the region replication stuff. After
> > > > >> fixing
> > > > >> > > this
> > > > >> > > > > > > > issue,
> > > > >> > > > > > > > > > > the only blocker is a stable 10B ITBLL run.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Hope we could get 3.0.0 out soon.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Thanks.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年3月6日周四
> > > > >> > > 22:55写道：
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > I've finished one round of rolling upgrading test.
> > > > >> It
> > > > >> > > worked
> > > > >> > > > > > fine.
> > > > >> > > > > > > > > > > > Please see HBASE-29161.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > But soon I found a problem when memstore
> > > > >> replication is
> > > > >> > > > > > enabled,
> > > > >> > > > > > > > and
> > > > >> > > > > > > > > > > > filed HBASE-29169.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > I will test upgrading with cluster replication too,
> > > > >> to
> > > > >> > > see if
> > > > >> > > > > > there
> > > > >> > > > > > > > > > > > are any other issues.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Thanks.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Istvan Toth <st...@apache.org> 于2025年1月20日周一
> > > > >> 14:24写道：
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > For HBASE-28431, only the final step of adding the
> > > > >> > > > > > Hadoop-less
> > > > >> > > > > > > > > assembly is
> > > > >> > > > > > > > > > > > > missing.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Please review
> > > > >> > > https://github.com/apache/hbase/pull/5766 if
> > > > >> > > > > > you
> > > > >> > > > > > > > > have the
> > > > >> > > > > > > > > > > > > time.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Istvan
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) <
> > > > >> > > > > > > > > palomino219@gmail.com> wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > I've finished the first round of issue
> > > > >> alignment, it
> > > > >> > > took
> > > > >> > > > > > > > > several months...
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > where we can have several ignore files, so we
> > > > >> do not
> > > > >> > > need
> > > > >> > > > > > to
> > > > >> > > > > > > > > manually
> > > > >> > > > > > > > > > > > > > check the result every time.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > So now, there are still several things to be
> > > > >> done
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > 1. Make sure we have all the features and fixes
> > > > >> we
> > > > >> > > want to
> > > > >> > > > > > land
> > > > >> > > > > > > > > on
> > > > >> > > > > > > > > > > > > > branch-3. For example, I think Istvan's assembly
> > > > >> > > cleanup
> > > > >> > > > > > should
> > > > >> > > > > > > > > be
> > > > >> > > > > > > > > > > > > > included. Please report other issues you want to
> > > > >> > > land in
> > > > >> > > > > > final
> > > > >> > > > > > > > > 3.0.0.
> > > > >> > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to
> > > > >> see
> > > > >> > > if I
> > > > >> > > > > > can
> > > > >> > > > > > > > get
> > > > >> > > > > > > > > > > > > > enough resources but it will be great if others
> > > > >> > > could help
> > > > >> > > > > > > > here.
> > > > >> > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling
> > > > >> > > upgrade
> > > > >> > > > > > test
> > > > >> > > > > > > > > with the
> > > > >> > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have
> > > > >> big
> > > > >> > > changes
> > > > >> > > > > > on
> > > > >> > > > > > > > > > > > > > replication and region replica replication, I
> > > > >> plan
> > > > >> > > to have
> > > > >> > > > > > > > tests
> > > > >> > > > > > > > > with
> > > > >> > > > > > > > > > > > > > these two features enabled too if possible.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Any other suggestions?
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Thanks.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> > > > >> 于2024年9月19日周四
> > > > >> > > > > > 22:35写道：
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > And do we want to apply the same trick to
> > > > >> > > MasterObserver
> > > > >> > > > > > and
> > > > >> > > > > > > > > > > > > > > RegionServerObserver?
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > For RegionObserver, we'd better do this
> > > > >> because
> > > > >> > > there are
> > > > >> > > > > > > > > already
> > > > >> > > > > > > > > > > > > > > methods in it that use the ObserverContext<?
> > > > >> > > extends
> > > > >> > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration,
> > > > >> which
> > > > >> > > were
> > > > >> > > > > > added
> > > > >> > > > > > > > in
> > > > >> > > > > > > > > > > > > > > HBASE-10378.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > For other coprocessor observers, we do not
> > > > >> have
> > > > >> > > such
> > > > >> > > > > > methods
> > > > >> > > > > > > > > yet.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> > > > >> 于2024年9月19日周四
> > > > >> > > > > > 22:03写道：
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm
> > > > >> inclined
> > > > >> > > toward
> > > > >> > > > > > doing
> > > > >> > > > > > > > > it here
> > > > >> > > > > > > > > > > > > > > > because (i think) we expect coprocessor
> > > > >> > > > > > implementations to
> > > > >> > > > > > > > > necessarily
> > > > >> > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not
> > > > >> thought
> > > > >> > > > > > through
> > > > >> > > > > > > > > what
> > > > >> > > > > > > > > > > > > > features are
> > > > >> > > > > > > > > > > > > > > > enabled by this new generic signature.
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo
> > > > >> Zhang) <
> > > > >> > > > > > > > > palomino219@gmail.com>
> > > > >> > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Filed HBASE-28862.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > It will break old coprocessor
> > > > >> implementations
> > > > >> > > even if
> > > > >> > > > > > > > they
> > > > >> > > > > > > > > do not use
> > > > >> > > > > > > > > > > > > > > > > any deprecated method. But since it does
> > > > >> not
> > > > >> > > change
> > > > >> > > > > > any
> > > > >> > > > > > > > > abilities, I
> > > > >> > > > > > > > > > > > > > > > > think it is OK to include this in a major
> > > > >> > > release as
> > > > >> > > > > > we
> > > > >> > > > > > > > > have already
> > > > >> > > > > > > > > > > > > > > > > done bunch of breaking changes, like
> > > > >> changing
> > > > >> > > the
> > > > >> > > > > > > > protobuf
> > > > >> > > > > > > > > from
> > > > >> > > > > > > > > > > > > > > > > unshaded one to shaded and relocated
> > > > >> one...
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > Thanks.
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> > > > >> > > 于2024年9月19日周四
> > > > >> > > > > > > > > 17:36写道：
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out.
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me
> > > > >> I
> > > > >> > > think we
> > > > >> > > > > > > > > should do this
> > > > >> > > > > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > > > > 3.0.0.
> > > > >> > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> > > > >> > > 于2024年9月19日周四
> > > > >> > > > > > > > > 16:46写道：
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > Should we take the opportunity to
> > > > >> upgrade
> > > > >> > > the
> > > > >> > > > > > generic
> > > > >> > > > > > > > > > > > > > definitions in
> > > > >> > > > > > > > > > > > > > > > > our
> > > > >> > > > > > > > > > > > > > > > > > > coprocessor signatures as well?
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > >
> > > > >> > >
> > > > >> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo
> > > > >> Zhang <
> > > > >> > > > > > > > > zhangduo@apache.org>
> > > > >> > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > I've done several rounds of API
> > > > >> cleanup
> > > > >> > > in
> > > > >> > > > > > > > > HBASE-24888 and now
> > > > >> > > > > > > > > > > > > > for me
> > > > >> > > > > > > > > > > > > > > > > > > > I think it is enough for making the
> > > > >> new
> > > > >> > > major
> > > > >> > > > > > > > > release.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning
> > > > >> the
> > > > >> > > commit
> > > > >> > > > > > > > > history and jira
> > > > >> > > > > > > > > > > > > > > > > > > > issues, it will be a huge project
> > > > >> for a
> > > > >> > > major
> > > > >> > > > > > > > > release and may
> > > > >> > > > > > > > > > > > > > take
> > > > >> > > > > > > > > > > > > > > > > > > > several weeks.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > After that, I think we are good to
> > > > >> go.
> > > > >> > > And I
> > > > >> > > > > > will
> > > > >> > > > > > > > > also try to
> > > > >> > > > > > > > > > > > > > find
> > > > >> > > > > > > > > > > > > > > > > > > > some resources to run several
> > > > >> rounds of
> > > > >> > > ITBLL
> > > > >> > > > > > and
> > > > >> > > > > > > > > also some
> > > > >> > > > > > > > > > > > > > upgrading
> > > > >> > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is
> > > > >> OK,
> > > > >> > > I will
> > > > >> > > > > > > > > release
> > > > >> > > > > > > > > > > > > > 3.0.0-beta-2,
> > > > >> > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the
> > > > >> > > 3.0.0-beta-2
> > > > >> > > > > > tag
> > > > >> > > > > > > > > and make the
> > > > >> > > > > > > > > > > > > > 3.0.0
> > > > >> > > > > > > > > > > > > > > > > > > > release soon.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > Hope I can get all these things done
> > > > >> > > within the
> > > > >> > > > > > > > year
> > > > >> > > > > > > > > 2024.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > > > > Thanks.
> > > > >> > > > > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > --
> > > > >> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > >> > > > > > > > *Email*: stoty@cloudera.com
> > > > >> > > > > > > > cloudera.com <https://www.cloudera.com>
> > > > >> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > >> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > > > >> > > [image:
> > > > >> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > > > >> > > [image:
> > > > >> > > > > > Cloudera
> > > > >> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > >> > > > > > > > ------------------------------
> > > > >> > > > > > > > ------------------------------
> > > > >> > > > > > > >
> > > > >> > > > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > *István Tóth* | Sr. Staff Software Engineer
> > > > >> > *Email*: stoty@cloudera.com
> > > > >> > cloudera.com <https://www.cloudera.com>
> > > > >> > [image: Cloudera] <https://www.cloudera.com/>
> > > > >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > >> Cloudera
> > > > >> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > >> > ------------------------------
> > > > >> > ------------------------------
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > *István Tóth* | Sr. Staff Software Engineer
> > > > > *Email*: stoty@cloudera.com
> > > > > cloudera.com <https://www.cloudera.com>
> > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > ------------------------------
> > > > > ------------------------------
> > > > >
> > > >
> > > >
> > > > --
> > > > *István Tóth* | Sr. Staff Software Engineer
> > > > *Email*: stoty@cloudera.com
> > > > cloudera.com <https://www.cloudera.com>
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > ------------------------------
> > > > ------------------------------

Re: [DISCUSS] For 3.0.0-beta-2 and the final 3.0.0 release

["张铎(Duo Zhang)" <pa...@gmail.com>]

The JUnit5 migration is nearly done.

I will start new rounds of 10B ITBLL to verify the stability of branch-3.

After merging the last several PRs for migration and the ITBLL result
is good, I will start cutting the 3.0.0-beta-2 release.

Thanks.

张铎(Duo Zhang) <pa...@gmail.com> 于2026年3月30日周一 19:28写道：
>
> Some updates here, there is no blocker issue for 3.0.0-beta-2 release
> now, but the junit5 upgrading is in its last stage, where we want to
> migrate all the tests in hbase-server, so I plan to finish the
> migration before cutting the final release.
>
> I will also start to run several rounds of 10B ITBLL to check the
> stability of branch-3.
>
> Thanks.
>
> 张铎(Duo Zhang) <pa...@gmail.com> 于2026年1月22日周四 19:15写道：
> >
> > Thanks Istvan for the detailed analysis!
> >
> > For HBASE-29834, I've already put up a PR to fix the problem. See
> >
> > https://github.com/apache/hbase/pull/7660
> >
> > And on HBASE-29843, I do not fully understand the problem and need
> > more context about the usage on Phoenix side. I think we should be
> > able to fix it.
> >
> > Thanks.
> >
> > Istvan Toth <st...@cloudera.com> 于2026年1月22日周四 14:33写道：
> > >
> > > This seems to have been caused by forward synchronous RPC changes between
> > > Hbase 2 and 3 (where HBase 2 simulates non-inclusive start keys by
> > > adding a \00 byte and setting them as inclusive, while HBase 3 handles them
> > > natively.
> > > This is something that we can handle on the Phoenix side.
> > >
> > > On the other hand, I've found a major and a minor breaking change in HBase
> > > 3:
> > >
> > > The big problem is HBASE-29843
> > > <https://issues.apache.org/jira/browse/HBASE-29843>
> > > HBaseRPCController.setPriority(final TableName tn) is never called on
> > > branch-3+
> > > Phoenix relies on setting it its own HBaseRPCController to set custom
> > > priorities, but HBase 3 has mostly abandoned using HBaseRPCController,
> > > leaving no
> > > working mechanism for Phoenix to use.
> > > HBase needs to restore using it to maintain compatibility. (or provide some
> > > alternative method)
> > >
> > > The smaller problem is HBASE-29834
> > > <https://issues.apache.org/jira/browse/HBASE-29834> MILLIS_BETWEEN_NEXTS
> > > metric is not updated on branch-3+
> > > This does not break Phoenix, it just happens to be detected by a Phoenix
> > > test.
> > >
> > > Istvan
> > >
> > >
> > > On Mon, Nov 3, 2025 at 10:22 AM Istvan Toth <st...@cloudera.com> wrote:
> > >
> > > > Sorry, I missed your reply eralier, Duo.
> > > >
> > > > I still haven't figured out the issue (though at least now I am now able
> > > > to run those tests from a debugger)
> > > > No ticket yet, as I'm still not sure which project to use.
> > > >
> > > > To repro this, we need to
> > > > * install a current HBase 3 snapshot locally from branch-3 head,
> > > > * check out my WIP Phoenix HBase 3 branch:
> > > > https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025
> > > > * then run a faling test:
> > > >
> > > > *mvn clean verify -am -pl phoenix-core
> > > > -Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray
> > > > -Dhbase.profile=3.0  -Dtest=foo*
> > > > There are other similar tests failures, missing expected ResultSet rows
> > > > after aggressively moving around the table regions during the query.
> > > >
> > > > Istvan
> > > >
> > > > On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > > > wrote:
> > > >
> > > >> Is there an issue in HBase or Phoenix for tracking this problem?
> > > >>
> > > >> Thanks.
> > > >>
> > > >> Istvan Toth <st...@cloudera.com.invalid> 于2025年10月16日周四 11:58写道：
> > > >> >
> > > >> > Quick heads up:
> > > >> > I'm investigating a regression in the Phoenix test suite where scans are
> > > >> > not reset when the processed region is moved, leading to missing data
> > > >> from
> > > >> > the scan.
> > > >> >
> > > >> > I am not positive yet that the issue is on the HBase side, will report
> > > >> here
> > > >> > (and open a ticket if needed) when I know more.
> > > >> >
> > > >> > Istvan
> > > >> >
> > > >> > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week to
> > > >> > > see if we can get these issues done, otherwise will move them out of
> > > >> > > 3.0.0.
> > > >> > >
> > > >> > > And the HBCK2 issue has been fixed.
> > > >> > >
> > > >> > > Plan to cut a new hbase-operator-tools release before cutting
> > > >> 3.0.0-beta-2.
> > > >> > >
> > > >> > > Thanks.
> > > >> > >
> > > >> > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年8月31日周日 22:57写道：
> > > >> > > >
> > > >> > > > I've finished one round of reviewing pending issues for
> > > >> 3.0.0-beta-2.
> > > >> > > >
> > > >> > > > There are still some necessary ones we need to get in before the
> > > >> final
> > > >> > > > 3.0.0 release, like make HBCK2 work against hbase3.
> > > >> > > >
> > > >> > > > Will keep working on them.
> > > >> > > >
> > > >> > > > Thanks.
> > > >> > > >
> > > >> > > > Nihal Jain <ni...@gmail.com> 于2025年6月19日周四 16:06写道：
> > > >> > > > >
> > > >> > > > > Yes, from hbase-thirdparty side, release is done and also
> > > >> consumed in
> > > >> > > hbase
> > > >> > > > > branch-2.5 and branch-2.6 and we are good here.
> > > >> > > > >
> > > >> > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) <
> > > >> palomino219@gmail.com>
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have
> > > >> already
> > > >> > > > > > made the new thirdparty release without jetty 12.
> > > >> > > > > >
> > > >> > > > > > Thanks.
> > > >> > > > > >
> > > >> > > > > > Nihal Jain <ni...@gmail.com> 于2025年6月18日周三 01:51写道：
> > > >> > > > > > >
> > > >> > > > > > > Ah yes, let me restart work for hbase thirdparty release for
> > > >> jetty
> > > >> > > 12
> > > >> > > > > > soon.
> > > >> > > > > > >
> > > >> > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth
> > > >> > > <stoty@cloudera.com.invalid
> > > >> > > > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > I think we should make sure we're using Jetty 12 in the next
> > > >> > > beta. (
> > > >> > > > > > so we
> > > >> > > > > > > > need a thirdparty release for Jetty 12)
> > > >> > > > > > > > We could also review other dependencies, and bump the ones
> > > >> (if
> > > >> > > any)
> > > >> > > > > > that
> > > >> > > > > > > > were kept on old versions because of the Java 8 requirement.
> > > >> > > > > > > >
> > > >> > > > > > > > Istvan
> > > >> > > > > > > >
> > > >> > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) <
> > > >> > > palomino219@gmail.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > The good news is that I've finally got a successful 10B
> > > >> ITBLL
> > > >> > > run.
> > > >> > > > > > > > >
> > > >> > > > > > > > > You can check HBASE-27919 for more details.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Next let me cleanup the jira issues and nightly runs.
> > > >> > > > > > > > >
> > > >> > > > > > > > > This time I think we can see 3.0.0 out soon.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Nihal Jain <ni...@apache.org> 于2025年3月12日周三 03:37写道：
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Hi Duo,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I
> > > >> encountered
> > > >> > > > > > HBASE-29144
> > > >> > > > > > > > > with kerberos. Posting here to just bring to attention.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Regards
> > > >> > > > > > > > > > Nihal
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote:
> > > >> > > > > > > > > > > A good news, cluster replication worked fine while
> > > >> rolling
> > > >> > > > > > upgrading
> > > >> > > > > > > > > in my test.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Let me focus on the region replication stuff. After
> > > >> fixing
> > > >> > > this
> > > >> > > > > > > > issue,
> > > >> > > > > > > > > > > the only blocker is a stable 10B ITBLL run.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Hope we could get 3.0.0 out soon.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年3月6日周四
> > > >> > > 22:55写道：
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > I've finished one round of rolling upgrading test.
> > > >> It
> > > >> > > worked
> > > >> > > > > > fine.
> > > >> > > > > > > > > > > > Please see HBASE-29161.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > But soon I found a problem when memstore
> > > >> replication is
> > > >> > > > > > enabled,
> > > >> > > > > > > > and
> > > >> > > > > > > > > > > > filed HBASE-29169.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > I will test upgrading with cluster replication too,
> > > >> to
> > > >> > > see if
> > > >> > > > > > there
> > > >> > > > > > > > > > > > are any other issues.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Thanks.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Istvan Toth <st...@apache.org> 于2025年1月20日周一
> > > >> 14:24写道：
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > For HBASE-28431, only the final step of adding the
> > > >> > > > > > Hadoop-less
> > > >> > > > > > > > > assembly is
> > > >> > > > > > > > > > > > > missing.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Please review
> > > >> > > https://github.com/apache/hbase/pull/5766 if
> > > >> > > > > > you
> > > >> > > > > > > > > have the
> > > >> > > > > > > > > > > > > time.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Istvan
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) <
> > > >> > > > > > > > > palomino219@gmail.com> wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > I've finished the first round of issue
> > > >> alignment, it
> > > >> > > took
> > > >> > > > > > > > > several months...
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > where we can have several ignore files, so we
> > > >> do not
> > > >> > > need
> > > >> > > > > > to
> > > >> > > > > > > > > manually
> > > >> > > > > > > > > > > > > > check the result every time.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > So now, there are still several things to be
> > > >> done
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > 1. Make sure we have all the features and fixes
> > > >> we
> > > >> > > want to
> > > >> > > > > > land
> > > >> > > > > > > > > on
> > > >> > > > > > > > > > > > > > branch-3. For example, I think Istvan's assembly
> > > >> > > cleanup
> > > >> > > > > > should
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > > > > > included. Please report other issues you want to
> > > >> > > land in
> > > >> > > > > > final
> > > >> > > > > > > > > 3.0.0.
> > > >> > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to
> > > >> see
> > > >> > > if I
> > > >> > > > > > can
> > > >> > > > > > > > get
> > > >> > > > > > > > > > > > > > enough resources but it will be great if others
> > > >> > > could help
> > > >> > > > > > > > here.
> > > >> > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling
> > > >> > > upgrade
> > > >> > > > > > test
> > > >> > > > > > > > > with the
> > > >> > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have
> > > >> big
> > > >> > > changes
> > > >> > > > > > on
> > > >> > > > > > > > > > > > > > replication and region replica replication, I
> > > >> plan
> > > >> > > to have
> > > >> > > > > > > > tests
> > > >> > > > > > > > > with
> > > >> > > > > > > > > > > > > > these two features enabled too if possible.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Any other suggestions?
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Thanks.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> > > >> 于2024年9月19日周四
> > > >> > > > > > 22:35写道：
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > And do we want to apply the same trick to
> > > >> > > MasterObserver
> > > >> > > > > > and
> > > >> > > > > > > > > > > > > > > RegionServerObserver?
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > For RegionObserver, we'd better do this
> > > >> because
> > > >> > > there are
> > > >> > > > > > > > > already
> > > >> > > > > > > > > > > > > > > methods in it that use the ObserverContext<?
> > > >> > > extends
> > > >> > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration,
> > > >> which
> > > >> > > were
> > > >> > > > > > added
> > > >> > > > > > > > in
> > > >> > > > > > > > > > > > > > > HBASE-10378.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > For other coprocessor observers, we do not
> > > >> have
> > > >> > > such
> > > >> > > > > > methods
> > > >> > > > > > > > > yet.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> > > >> 于2024年9月19日周四
> > > >> > > > > > 22:03写道：
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm
> > > >> inclined
> > > >> > > toward
> > > >> > > > > > doing
> > > >> > > > > > > > > it here
> > > >> > > > > > > > > > > > > > > > because (i think) we expect coprocessor
> > > >> > > > > > implementations to
> > > >> > > > > > > > > necessarily
> > > >> > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not
> > > >> thought
> > > >> > > > > > through
> > > >> > > > > > > > > what
> > > >> > > > > > > > > > > > > > features are
> > > >> > > > > > > > > > > > > > > > enabled by this new generic signature.
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo
> > > >> Zhang) <
> > > >> > > > > > > > > palomino219@gmail.com>
> > > >> > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Filed HBASE-28862.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > It will break old coprocessor
> > > >> implementations
> > > >> > > even if
> > > >> > > > > > > > they
> > > >> > > > > > > > > do not use
> > > >> > > > > > > > > > > > > > > > > any deprecated method. But since it does
> > > >> not
> > > >> > > change
> > > >> > > > > > any
> > > >> > > > > > > > > abilities, I
> > > >> > > > > > > > > > > > > > > > > think it is OK to include this in a major
> > > >> > > release as
> > > >> > > > > > we
> > > >> > > > > > > > > have already
> > > >> > > > > > > > > > > > > > > > > done bunch of breaking changes, like
> > > >> changing
> > > >> > > the
> > > >> > > > > > > > protobuf
> > > >> > > > > > > > > from
> > > >> > > > > > > > > > > > > > > > > unshaded one to shaded and relocated
> > > >> one...
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Thanks.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> > > >> > > 于2024年9月19日周四
> > > >> > > > > > > > > 17:36写道：
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me
> > > >> I
> > > >> > > think we
> > > >> > > > > > > > > should do this
> > > >> > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > 3.0.0.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> > > >> > > 于2024年9月19日周四
> > > >> > > > > > > > > 16:46写道：
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > Should we take the opportunity to
> > > >> upgrade
> > > >> > > the
> > > >> > > > > > generic
> > > >> > > > > > > > > > > > > > definitions in
> > > >> > > > > > > > > > > > > > > > > our
> > > >> > > > > > > > > > > > > > > > > > > coprocessor signatures as well?
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > >
> > > >> > >
> > > >> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo
> > > >> Zhang <
> > > >> > > > > > > > > zhangduo@apache.org>
> > > >> > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > I've done several rounds of API
> > > >> cleanup
> > > >> > > in
> > > >> > > > > > > > > HBASE-24888 and now
> > > >> > > > > > > > > > > > > > for me
> > > >> > > > > > > > > > > > > > > > > > > > I think it is enough for making the
> > > >> new
> > > >> > > major
> > > >> > > > > > > > > release.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning
> > > >> the
> > > >> > > commit
> > > >> > > > > > > > > history and jira
> > > >> > > > > > > > > > > > > > > > > > > > issues, it will be a huge project
> > > >> for a
> > > >> > > major
> > > >> > > > > > > > > release and may
> > > >> > > > > > > > > > > > > > take
> > > >> > > > > > > > > > > > > > > > > > > > several weeks.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > After that, I think we are good to
> > > >> go.
> > > >> > > And I
> > > >> > > > > > will
> > > >> > > > > > > > > also try to
> > > >> > > > > > > > > > > > > > find
> > > >> > > > > > > > > > > > > > > > > > > > some resources to run several
> > > >> rounds of
> > > >> > > ITBLL
> > > >> > > > > > and
> > > >> > > > > > > > > also some
> > > >> > > > > > > > > > > > > > upgrading
> > > >> > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is
> > > >> OK,
> > > >> > > I will
> > > >> > > > > > > > > release
> > > >> > > > > > > > > > > > > > 3.0.0-beta-2,
> > > >> > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the
> > > >> > > 3.0.0-beta-2
> > > >> > > > > > tag
> > > >> > > > > > > > > and make the
> > > >> > > > > > > > > > > > > > 3.0.0
> > > >> > > > > > > > > > > > > > > > > > > > release soon.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > Hope I can get all these things done
> > > >> > > within the
> > > >> > > > > > > > year
> > > >> > > > > > > > > 2024.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > Thanks.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> > > >> > > > > > > > *Email*: stoty@cloudera.com
> > > >> > > > > > > > cloudera.com <https://www.cloudera.com>
> > > >> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > >> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > > >> > > [image:
> > > >> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > > >> > > [image:
> > > >> > > > > > Cloudera
> > > >> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > >> > > > > > > > ------------------------------
> > > >> > > > > > > > ------------------------------
> > > >> > > > > > > >
> > > >> > > > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > *István Tóth* | Sr. Staff Software Engineer
> > > >> > *Email*: stoty@cloudera.com
> > > >> > cloudera.com <https://www.cloudera.com>
> > > >> > [image: Cloudera] <https://www.cloudera.com/>
> > > >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > >> Cloudera
> > > >> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > >> > ------------------------------
> > > >> > ------------------------------
> > > >>
> > > >
> > > >
> > > > --
> > > > *István Tóth* | Sr. Staff Software Engineer
> > > > *Email*: stoty@cloudera.com
> > > > cloudera.com <https://www.cloudera.com>
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > ------------------------------
> > > > ------------------------------
> > > >
> > >
> > >
> > > --
> > > *István Tóth* | Sr. Staff Software Engineer
> > > *Email*: stoty@cloudera.com
> > > cloudera.com <https://www.cloudera.com>
> > > [image: Cloudera] <https://www.cloudera.com/>
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > ------------------------------
> > > ------------------------------

Re: [DISCUSS] For 3.0.0-beta-2 and the final 3.0.0 release

["张铎(Duo Zhang)" <pa...@gmail.com>]

Some updates here, there is no blocker issue for 3.0.0-beta-2 release
now, but the junit5 upgrading is in its last stage, where we want to
migrate all the tests in hbase-server, so I plan to finish the
migration before cutting the final release.

I will also start to run several rounds of 10B ITBLL to check the
stability of branch-3.

Thanks.

张铎(Duo Zhang) <pa...@gmail.com> 于2026年1月22日周四 19:15写道：
>
> Thanks Istvan for the detailed analysis!
>
> For HBASE-29834, I've already put up a PR to fix the problem. See
>
> https://github.com/apache/hbase/pull/7660
>
> And on HBASE-29843, I do not fully understand the problem and need
> more context about the usage on Phoenix side. I think we should be
> able to fix it.
>
> Thanks.
>
> Istvan Toth <st...@cloudera.com> 于2026年1月22日周四 14:33写道：
> >
> > This seems to have been caused by forward synchronous RPC changes between
> > Hbase 2 and 3 (where HBase 2 simulates non-inclusive start keys by
> > adding a \00 byte and setting them as inclusive, while HBase 3 handles them
> > natively.
> > This is something that we can handle on the Phoenix side.
> >
> > On the other hand, I've found a major and a minor breaking change in HBase
> > 3:
> >
> > The big problem is HBASE-29843
> > <https://issues.apache.org/jira/browse/HBASE-29843>
> > HBaseRPCController.setPriority(final TableName tn) is never called on
> > branch-3+
> > Phoenix relies on setting it its own HBaseRPCController to set custom
> > priorities, but HBase 3 has mostly abandoned using HBaseRPCController,
> > leaving no
> > working mechanism for Phoenix to use.
> > HBase needs to restore using it to maintain compatibility. (or provide some
> > alternative method)
> >
> > The smaller problem is HBASE-29834
> > <https://issues.apache.org/jira/browse/HBASE-29834> MILLIS_BETWEEN_NEXTS
> > metric is not updated on branch-3+
> > This does not break Phoenix, it just happens to be detected by a Phoenix
> > test.
> >
> > Istvan
> >
> >
> > On Mon, Nov 3, 2025 at 10:22 AM Istvan Toth <st...@cloudera.com> wrote:
> >
> > > Sorry, I missed your reply eralier, Duo.
> > >
> > > I still haven't figured out the issue (though at least now I am now able
> > > to run those tests from a debugger)
> > > No ticket yet, as I'm still not sure which project to use.
> > >
> > > To repro this, we need to
> > > * install a current HBase 3 snapshot locally from branch-3 head,
> > > * check out my WIP Phoenix HBase 3 branch:
> > > https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025
> > > * then run a faling test:
> > >
> > > *mvn clean verify -am -pl phoenix-core
> > > -Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray
> > > -Dhbase.profile=3.0  -Dtest=foo*
> > > There are other similar tests failures, missing expected ResultSet rows
> > > after aggressively moving around the table regions during the query.
> > >
> > > Istvan
> > >
> > > On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > > wrote:
> > >
> > >> Is there an issue in HBase or Phoenix for tracking this problem?
> > >>
> > >> Thanks.
> > >>
> > >> Istvan Toth <st...@cloudera.com.invalid> 于2025年10月16日周四 11:58写道：
> > >> >
> > >> > Quick heads up:
> > >> > I'm investigating a regression in the Phoenix test suite where scans are
> > >> > not reset when the processed region is moved, leading to missing data
> > >> from
> > >> > the scan.
> > >> >
> > >> > I am not positive yet that the issue is on the HBase side, will report
> > >> here
> > >> > (and open a ticket if needed) when I know more.
> > >> >
> > >> > Istvan
> > >> >
> > >> > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week to
> > >> > > see if we can get these issues done, otherwise will move them out of
> > >> > > 3.0.0.
> > >> > >
> > >> > > And the HBCK2 issue has been fixed.
> > >> > >
> > >> > > Plan to cut a new hbase-operator-tools release before cutting
> > >> 3.0.0-beta-2.
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年8月31日周日 22:57写道：
> > >> > > >
> > >> > > > I've finished one round of reviewing pending issues for
> > >> 3.0.0-beta-2.
> > >> > > >
> > >> > > > There are still some necessary ones we need to get in before the
> > >> final
> > >> > > > 3.0.0 release, like make HBCK2 work against hbase3.
> > >> > > >
> > >> > > > Will keep working on them.
> > >> > > >
> > >> > > > Thanks.
> > >> > > >
> > >> > > > Nihal Jain <ni...@gmail.com> 于2025年6月19日周四 16:06写道：
> > >> > > > >
> > >> > > > > Yes, from hbase-thirdparty side, release is done and also
> > >> consumed in
> > >> > > hbase
> > >> > > > > branch-2.5 and branch-2.6 and we are good here.
> > >> > > > >
> > >> > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) <
> > >> palomino219@gmail.com>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have
> > >> already
> > >> > > > > > made the new thirdparty release without jetty 12.
> > >> > > > > >
> > >> > > > > > Thanks.
> > >> > > > > >
> > >> > > > > > Nihal Jain <ni...@gmail.com> 于2025年6月18日周三 01:51写道：
> > >> > > > > > >
> > >> > > > > > > Ah yes, let me restart work for hbase thirdparty release for
> > >> jetty
> > >> > > 12
> > >> > > > > > soon.
> > >> > > > > > >
> > >> > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth
> > >> > > <stoty@cloudera.com.invalid
> > >> > > > > > >
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > I think we should make sure we're using Jetty 12 in the next
> > >> > > beta. (
> > >> > > > > > so we
> > >> > > > > > > > need a thirdparty release for Jetty 12)
> > >> > > > > > > > We could also review other dependencies, and bump the ones
> > >> (if
> > >> > > any)
> > >> > > > > > that
> > >> > > > > > > > were kept on old versions because of the Java 8 requirement.
> > >> > > > > > > >
> > >> > > > > > > > Istvan
> > >> > > > > > > >
> > >> > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) <
> > >> > > palomino219@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > The good news is that I've finally got a successful 10B
> > >> ITBLL
> > >> > > run.
> > >> > > > > > > > >
> > >> > > > > > > > > You can check HBASE-27919 for more details.
> > >> > > > > > > > >
> > >> > > > > > > > > Next let me cleanup the jira issues and nightly runs.
> > >> > > > > > > > >
> > >> > > > > > > > > This time I think we can see 3.0.0 out soon.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks.
> > >> > > > > > > > >
> > >> > > > > > > > > Nihal Jain <ni...@apache.org> 于2025年3月12日周三 03:37写道：
> > >> > > > > > > > > >
> > >> > > > > > > > > > Hi Duo,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I
> > >> encountered
> > >> > > > > > HBASE-29144
> > >> > > > > > > > > with kerberos. Posting here to just bring to attention.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Regards
> > >> > > > > > > > > > Nihal
> > >> > > > > > > > > >
> > >> > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote:
> > >> > > > > > > > > > > A good news, cluster replication worked fine while
> > >> rolling
> > >> > > > > > upgrading
> > >> > > > > > > > > in my test.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Let me focus on the region replication stuff. After
> > >> fixing
> > >> > > this
> > >> > > > > > > > issue,
> > >> > > > > > > > > > > the only blocker is a stable 10B ITBLL run.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Hope we could get 3.0.0 out soon.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年3月6日周四
> > >> > > 22:55写道：
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > I've finished one round of rolling upgrading test.
> > >> It
> > >> > > worked
> > >> > > > > > fine.
> > >> > > > > > > > > > > > Please see HBASE-29161.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > But soon I found a problem when memstore
> > >> replication is
> > >> > > > > > enabled,
> > >> > > > > > > > and
> > >> > > > > > > > > > > > filed HBASE-29169.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > I will test upgrading with cluster replication too,
> > >> to
> > >> > > see if
> > >> > > > > > there
> > >> > > > > > > > > > > > are any other issues.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Istvan Toth <st...@apache.org> 于2025年1月20日周一
> > >> 14:24写道：
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > For HBASE-28431, only the final step of adding the
> > >> > > > > > Hadoop-less
> > >> > > > > > > > > assembly is
> > >> > > > > > > > > > > > > missing.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Please review
> > >> > > https://github.com/apache/hbase/pull/5766 if
> > >> > > > > > you
> > >> > > > > > > > > have the
> > >> > > > > > > > > > > > > time.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Istvan
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) <
> > >> > > > > > > > > palomino219@gmail.com> wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I've finished the first round of issue
> > >> alignment, it
> > >> > > took
> > >> > > > > > > > > several months...
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > where we can have several ignore files, so we
> > >> do not
> > >> > > need
> > >> > > > > > to
> > >> > > > > > > > > manually
> > >> > > > > > > > > > > > > > check the result every time.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > So now, there are still several things to be
> > >> done
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > 1. Make sure we have all the features and fixes
> > >> we
> > >> > > want to
> > >> > > > > > land
> > >> > > > > > > > > on
> > >> > > > > > > > > > > > > > branch-3. For example, I think Istvan's assembly
> > >> > > cleanup
> > >> > > > > > should
> > >> > > > > > > > > be
> > >> > > > > > > > > > > > > > included. Please report other issues you want to
> > >> > > land in
> > >> > > > > > final
> > >> > > > > > > > > 3.0.0.
> > >> > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to
> > >> see
> > >> > > if I
> > >> > > > > > can
> > >> > > > > > > > get
> > >> > > > > > > > > > > > > > enough resources but it will be great if others
> > >> > > could help
> > >> > > > > > > > here.
> > >> > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling
> > >> > > upgrade
> > >> > > > > > test
> > >> > > > > > > > > with the
> > >> > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have
> > >> big
> > >> > > changes
> > >> > > > > > on
> > >> > > > > > > > > > > > > > replication and region replica replication, I
> > >> plan
> > >> > > to have
> > >> > > > > > > > tests
> > >> > > > > > > > > with
> > >> > > > > > > > > > > > > > these two features enabled too if possible.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Any other suggestions?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Thanks.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> > >> 于2024年9月19日周四
> > >> > > > > > 22:35写道：
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > And do we want to apply the same trick to
> > >> > > MasterObserver
> > >> > > > > > and
> > >> > > > > > > > > > > > > > > RegionServerObserver?
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > For RegionObserver, we'd better do this
> > >> because
> > >> > > there are
> > >> > > > > > > > > already
> > >> > > > > > > > > > > > > > > methods in it that use the ObserverContext<?
> > >> > > extends
> > >> > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration,
> > >> which
> > >> > > were
> > >> > > > > > added
> > >> > > > > > > > in
> > >> > > > > > > > > > > > > > > HBASE-10378.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > For other coprocessor observers, we do not
> > >> have
> > >> > > such
> > >> > > > > > methods
> > >> > > > > > > > > yet.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> > >> 于2024年9月19日周四
> > >> > > > > > 22:03写道：
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm
> > >> inclined
> > >> > > toward
> > >> > > > > > doing
> > >> > > > > > > > > it here
> > >> > > > > > > > > > > > > > > > because (i think) we expect coprocessor
> > >> > > > > > implementations to
> > >> > > > > > > > > necessarily
> > >> > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not
> > >> thought
> > >> > > > > > through
> > >> > > > > > > > > what
> > >> > > > > > > > > > > > > > features are
> > >> > > > > > > > > > > > > > > > enabled by this new generic signature.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo
> > >> Zhang) <
> > >> > > > > > > > > palomino219@gmail.com>
> > >> > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Filed HBASE-28862.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > It will break old coprocessor
> > >> implementations
> > >> > > even if
> > >> > > > > > > > they
> > >> > > > > > > > > do not use
> > >> > > > > > > > > > > > > > > > > any deprecated method. But since it does
> > >> not
> > >> > > change
> > >> > > > > > any
> > >> > > > > > > > > abilities, I
> > >> > > > > > > > > > > > > > > > > think it is OK to include this in a major
> > >> > > release as
> > >> > > > > > we
> > >> > > > > > > > > have already
> > >> > > > > > > > > > > > > > > > > done bunch of breaking changes, like
> > >> changing
> > >> > > the
> > >> > > > > > > > protobuf
> > >> > > > > > > > > from
> > >> > > > > > > > > > > > > > > > > unshaded one to shaded and relocated
> > >> one...
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Thanks.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> > >> > > 于2024年9月19日周四
> > >> > > > > > > > > 17:36写道：
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me
> > >> I
> > >> > > think we
> > >> > > > > > > > > should do this
> > >> > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > 3.0.0.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> > >> > > 于2024年9月19日周四
> > >> > > > > > > > > 16:46写道：
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > Should we take the opportunity to
> > >> upgrade
> > >> > > the
> > >> > > > > > generic
> > >> > > > > > > > > > > > > > definitions in
> > >> > > > > > > > > > > > > > > > > our
> > >> > > > > > > > > > > > > > > > > > > coprocessor signatures as well?
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > >
> > >> > >
> > >> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo
> > >> Zhang <
> > >> > > > > > > > > zhangduo@apache.org>
> > >> > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > I've done several rounds of API
> > >> cleanup
> > >> > > in
> > >> > > > > > > > > HBASE-24888 and now
> > >> > > > > > > > > > > > > > for me
> > >> > > > > > > > > > > > > > > > > > > > I think it is enough for making the
> > >> new
> > >> > > major
> > >> > > > > > > > > release.
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning
> > >> the
> > >> > > commit
> > >> > > > > > > > > history and jira
> > >> > > > > > > > > > > > > > > > > > > > issues, it will be a huge project
> > >> for a
> > >> > > major
> > >> > > > > > > > > release and may
> > >> > > > > > > > > > > > > > take
> > >> > > > > > > > > > > > > > > > > > > > several weeks.
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > After that, I think we are good to
> > >> go.
> > >> > > And I
> > >> > > > > > will
> > >> > > > > > > > > also try to
> > >> > > > > > > > > > > > > > find
> > >> > > > > > > > > > > > > > > > > > > > some resources to run several
> > >> rounds of
> > >> > > ITBLL
> > >> > > > > > and
> > >> > > > > > > > > also some
> > >> > > > > > > > > > > > > > upgrading
> > >> > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is
> > >> OK,
> > >> > > I will
> > >> > > > > > > > > release
> > >> > > > > > > > > > > > > > 3.0.0-beta-2,
> > >> > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the
> > >> > > 3.0.0-beta-2
> > >> > > > > > tag
> > >> > > > > > > > > and make the
> > >> > > > > > > > > > > > > > 3.0.0
> > >> > > > > > > > > > > > > > > > > > > > release soon.
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > Hope I can get all these things done
> > >> > > within the
> > >> > > > > > > > year
> > >> > > > > > > > > 2024.
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > Thanks.
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> > >> > > > > > > > *Email*: stoty@cloudera.com
> > >> > > > > > > > cloudera.com <https://www.cloudera.com>
> > >> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > >> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > >> > > [image:
> > >> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > >> > > [image:
> > >> > > > > > Cloudera
> > >> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > >> > > > > > > > ------------------------------
> > >> > > > > > > > ------------------------------
> > >> > > > > > > >
> > >> > > > > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > *István Tóth* | Sr. Staff Software Engineer
> > >> > *Email*: stoty@cloudera.com
> > >> > cloudera.com <https://www.cloudera.com>
> > >> > [image: Cloudera] <https://www.cloudera.com/>
> > >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > >> Cloudera
> > >> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > >> > ------------------------------
> > >> > ------------------------------
> > >>
> > >
> > >
> > > --
> > > *István Tóth* | Sr. Staff Software Engineer
> > > *Email*: stoty@cloudera.com
> > > cloudera.com <https://www.cloudera.com>
> > > [image: Cloudera] <https://www.cloudera.com/>
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > ------------------------------
> > > ------------------------------
> > >
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------

Re: [DISCUSS] For 3.0.0-beta-2 and the final 3.0.0 release

["张铎(Duo Zhang)" <pa...@gmail.com>]

Thanks Istvan for the detailed analysis!

For HBASE-29834, I've already put up a PR to fix the problem. See

https://github.com/apache/hbase/pull/7660

And on HBASE-29843, I do not fully understand the problem and need
more context about the usage on Phoenix side. I think we should be
able to fix it.

Thanks.

Istvan Toth <st...@cloudera.com> 于2026年1月22日周四 14:33写道：
>
> This seems to have been caused by forward synchronous RPC changes between
> Hbase 2 and 3 (where HBase 2 simulates non-inclusive start keys by
> adding a \00 byte and setting them as inclusive, while HBase 3 handles them
> natively.
> This is something that we can handle on the Phoenix side.
>
> On the other hand, I've found a major and a minor breaking change in HBase
> 3:
>
> The big problem is HBASE-29843
> <https://issues.apache.org/jira/browse/HBASE-29843>
> HBaseRPCController.setPriority(final TableName tn) is never called on
> branch-3+
> Phoenix relies on setting it its own HBaseRPCController to set custom
> priorities, but HBase 3 has mostly abandoned using HBaseRPCController,
> leaving no
> working mechanism for Phoenix to use.
> HBase needs to restore using it to maintain compatibility. (or provide some
> alternative method)
>
> The smaller problem is HBASE-29834
> <https://issues.apache.org/jira/browse/HBASE-29834> MILLIS_BETWEEN_NEXTS
> metric is not updated on branch-3+
> This does not break Phoenix, it just happens to be detected by a Phoenix
> test.
>
> Istvan
>
>
> On Mon, Nov 3, 2025 at 10:22 AM Istvan Toth <st...@cloudera.com> wrote:
>
> > Sorry, I missed your reply eralier, Duo.
> >
> > I still haven't figured out the issue (though at least now I am now able
> > to run those tests from a debugger)
> > No ticket yet, as I'm still not sure which project to use.
> >
> > To repro this, we need to
> > * install a current HBase 3 snapshot locally from branch-3 head,
> > * check out my WIP Phoenix HBase 3 branch:
> > https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025
> > * then run a faling test:
> >
> > *mvn clean verify -am -pl phoenix-core
> > -Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray
> > -Dhbase.profile=3.0  -Dtest=foo*
> > There are other similar tests failures, missing expected ResultSet rows
> > after aggressively moving around the table regions during the query.
> >
> > Istvan
> >
> > On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> >> Is there an issue in HBase or Phoenix for tracking this problem?
> >>
> >> Thanks.
> >>
> >> Istvan Toth <st...@cloudera.com.invalid> 于2025年10月16日周四 11:58写道：
> >> >
> >> > Quick heads up:
> >> > I'm investigating a regression in the Phoenix test suite where scans are
> >> > not reset when the processed region is moved, leading to missing data
> >> from
> >> > the scan.
> >> >
> >> > I am not positive yet that the issue is on the HBase side, will report
> >> here
> >> > (and open a ticket if needed) when I know more.
> >> >
> >> > Istvan
> >> >
> >> > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <pa...@gmail.com>
> >> wrote:
> >> >
> >> > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week to
> >> > > see if we can get these issues done, otherwise will move them out of
> >> > > 3.0.0.
> >> > >
> >> > > And the HBCK2 issue has been fixed.
> >> > >
> >> > > Plan to cut a new hbase-operator-tools release before cutting
> >> 3.0.0-beta-2.
> >> > >
> >> > > Thanks.
> >> > >
> >> > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年8月31日周日 22:57写道：
> >> > > >
> >> > > > I've finished one round of reviewing pending issues for
> >> 3.0.0-beta-2.
> >> > > >
> >> > > > There are still some necessary ones we need to get in before the
> >> final
> >> > > > 3.0.0 release, like make HBCK2 work against hbase3.
> >> > > >
> >> > > > Will keep working on them.
> >> > > >
> >> > > > Thanks.
> >> > > >
> >> > > > Nihal Jain <ni...@gmail.com> 于2025年6月19日周四 16:06写道：
> >> > > > >
> >> > > > > Yes, from hbase-thirdparty side, release is done and also
> >> consumed in
> >> > > hbase
> >> > > > > branch-2.5 and branch-2.6 and we are good here.
> >> > > > >
> >> > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) <
> >> palomino219@gmail.com>
> >> > > wrote:
> >> > > > >
> >> > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have
> >> already
> >> > > > > > made the new thirdparty release without jetty 12.
> >> > > > > >
> >> > > > > > Thanks.
> >> > > > > >
> >> > > > > > Nihal Jain <ni...@gmail.com> 于2025年6月18日周三 01:51写道：
> >> > > > > > >
> >> > > > > > > Ah yes, let me restart work for hbase thirdparty release for
> >> jetty
> >> > > 12
> >> > > > > > soon.
> >> > > > > > >
> >> > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth
> >> > > <stoty@cloudera.com.invalid
> >> > > > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > I think we should make sure we're using Jetty 12 in the next
> >> > > beta. (
> >> > > > > > so we
> >> > > > > > > > need a thirdparty release for Jetty 12)
> >> > > > > > > > We could also review other dependencies, and bump the ones
> >> (if
> >> > > any)
> >> > > > > > that
> >> > > > > > > > were kept on old versions because of the Java 8 requirement.
> >> > > > > > > >
> >> > > > > > > > Istvan
> >> > > > > > > >
> >> > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) <
> >> > > palomino219@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > The good news is that I've finally got a successful 10B
> >> ITBLL
> >> > > run.
> >> > > > > > > > >
> >> > > > > > > > > You can check HBASE-27919 for more details.
> >> > > > > > > > >
> >> > > > > > > > > Next let me cleanup the jira issues and nightly runs.
> >> > > > > > > > >
> >> > > > > > > > > This time I think we can see 3.0.0 out soon.
> >> > > > > > > > >
> >> > > > > > > > > Thanks.
> >> > > > > > > > >
> >> > > > > > > > > Nihal Jain <ni...@apache.org> 于2025年3月12日周三 03:37写道：
> >> > > > > > > > > >
> >> > > > > > > > > > Hi Duo,
> >> > > > > > > > > >
> >> > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I
> >> encountered
> >> > > > > > HBASE-29144
> >> > > > > > > > > with kerberos. Posting here to just bring to attention.
> >> > > > > > > > > >
> >> > > > > > > > > > Regards
> >> > > > > > > > > > Nihal
> >> > > > > > > > > >
> >> > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote:
> >> > > > > > > > > > > A good news, cluster replication worked fine while
> >> rolling
> >> > > > > > upgrading
> >> > > > > > > > > in my test.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Let me focus on the region replication stuff. After
> >> fixing
> >> > > this
> >> > > > > > > > issue,
> >> > > > > > > > > > > the only blocker is a stable 10B ITBLL run.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Hope we could get 3.0.0 out soon.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks.
> >> > > > > > > > > > >
> >> > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年3月6日周四
> >> > > 22:55写道：
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I've finished one round of rolling upgrading test.
> >> It
> >> > > worked
> >> > > > > > fine.
> >> > > > > > > > > > > > Please see HBASE-29161.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > But soon I found a problem when memstore
> >> replication is
> >> > > > > > enabled,
> >> > > > > > > > and
> >> > > > > > > > > > > > filed HBASE-29169.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I will test upgrading with cluster replication too,
> >> to
> >> > > see if
> >> > > > > > there
> >> > > > > > > > > > > > are any other issues.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Istvan Toth <st...@apache.org> 于2025年1月20日周一
> >> 14:24写道：
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > For HBASE-28431, only the final step of adding the
> >> > > > > > Hadoop-less
> >> > > > > > > > > assembly is
> >> > > > > > > > > > > > > missing.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Please review
> >> > > https://github.com/apache/hbase/pull/5766 if
> >> > > > > > you
> >> > > > > > > > > have the
> >> > > > > > > > > > > > > time.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Istvan
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) <
> >> > > > > > > > > palomino219@gmail.com> wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I've finished the first round of issue
> >> alignment, it
> >> > > took
> >> > > > > > > > > several months...
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > where we can have several ignore files, so we
> >> do not
> >> > > need
> >> > > > > > to
> >> > > > > > > > > manually
> >> > > > > > > > > > > > > > check the result every time.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > So now, there are still several things to be
> >> done
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > 1. Make sure we have all the features and fixes
> >> we
> >> > > want to
> >> > > > > > land
> >> > > > > > > > > on
> >> > > > > > > > > > > > > > branch-3. For example, I think Istvan's assembly
> >> > > cleanup
> >> > > > > > should
> >> > > > > > > > > be
> >> > > > > > > > > > > > > > included. Please report other issues you want to
> >> > > land in
> >> > > > > > final
> >> > > > > > > > > 3.0.0.
> >> > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to
> >> see
> >> > > if I
> >> > > > > > can
> >> > > > > > > > get
> >> > > > > > > > > > > > > > enough resources but it will be great if others
> >> > > could help
> >> > > > > > > > here.
> >> > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling
> >> > > upgrade
> >> > > > > > test
> >> > > > > > > > > with the
> >> > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have
> >> big
> >> > > changes
> >> > > > > > on
> >> > > > > > > > > > > > > > replication and region replica replication, I
> >> plan
> >> > > to have
> >> > > > > > > > tests
> >> > > > > > > > > with
> >> > > > > > > > > > > > > > these two features enabled too if possible.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Any other suggestions?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Thanks.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> >> 于2024年9月19日周四
> >> > > > > > 22:35写道：
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > And do we want to apply the same trick to
> >> > > MasterObserver
> >> > > > > > and
> >> > > > > > > > > > > > > > > RegionServerObserver?
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > For RegionObserver, we'd better do this
> >> because
> >> > > there are
> >> > > > > > > > > already
> >> > > > > > > > > > > > > > > methods in it that use the ObserverContext<?
> >> > > extends
> >> > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration,
> >> which
> >> > > were
> >> > > > > > added
> >> > > > > > > > in
> >> > > > > > > > > > > > > > > HBASE-10378.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > For other coprocessor observers, we do not
> >> have
> >> > > such
> >> > > > > > methods
> >> > > > > > > > > yet.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> >> 于2024年9月19日周四
> >> > > > > > 22:03写道：
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm
> >> inclined
> >> > > toward
> >> > > > > > doing
> >> > > > > > > > > it here
> >> > > > > > > > > > > > > > > > because (i think) we expect coprocessor
> >> > > > > > implementations to
> >> > > > > > > > > necessarily
> >> > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not
> >> thought
> >> > > > > > through
> >> > > > > > > > > what
> >> > > > > > > > > > > > > > features are
> >> > > > > > > > > > > > > > > > enabled by this new generic signature.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo
> >> Zhang) <
> >> > > > > > > > > palomino219@gmail.com>
> >> > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Filed HBASE-28862.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > It will break old coprocessor
> >> implementations
> >> > > even if
> >> > > > > > > > they
> >> > > > > > > > > do not use
> >> > > > > > > > > > > > > > > > > any deprecated method. But since it does
> >> not
> >> > > change
> >> > > > > > any
> >> > > > > > > > > abilities, I
> >> > > > > > > > > > > > > > > > > think it is OK to include this in a major
> >> > > release as
> >> > > > > > we
> >> > > > > > > > > have already
> >> > > > > > > > > > > > > > > > > done bunch of breaking changes, like
> >> changing
> >> > > the
> >> > > > > > > > protobuf
> >> > > > > > > > > from
> >> > > > > > > > > > > > > > > > > unshaded one to shaded and relocated
> >> one...
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Thanks.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
> >> > > 于2024年9月19日周四
> >> > > > > > > > > 17:36写道：
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me
> >> I
> >> > > think we
> >> > > > > > > > > should do this
> >> > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > 3.0.0.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
> >> > > 于2024年9月19日周四
> >> > > > > > > > > 16:46写道：
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > Should we take the opportunity to
> >> upgrade
> >> > > the
> >> > > > > > generic
> >> > > > > > > > > > > > > > definitions in
> >> > > > > > > > > > > > > > > > > our
> >> > > > > > > > > > > > > > > > > > > coprocessor signatures as well?
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > >
> >> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo
> >> Zhang <
> >> > > > > > > > > zhangduo@apache.org>
> >> > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > I've done several rounds of API
> >> cleanup
> >> > > in
> >> > > > > > > > > HBASE-24888 and now
> >> > > > > > > > > > > > > > for me
> >> > > > > > > > > > > > > > > > > > > > I think it is enough for making the
> >> new
> >> > > major
> >> > > > > > > > > release.
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning
> >> the
> >> > > commit
> >> > > > > > > > > history and jira
> >> > > > > > > > > > > > > > > > > > > > issues, it will be a huge project
> >> for a
> >> > > major
> >> > > > > > > > > release and may
> >> > > > > > > > > > > > > > take
> >> > > > > > > > > > > > > > > > > > > > several weeks.
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > After that, I think we are good to
> >> go.
> >> > > And I
> >> > > > > > will
> >> > > > > > > > > also try to
> >> > > > > > > > > > > > > > find
> >> > > > > > > > > > > > > > > > > > > > some resources to run several
> >> rounds of
> >> > > ITBLL
> >> > > > > > and
> >> > > > > > > > > also some
> >> > > > > > > > > > > > > > upgrading
> >> > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is
> >> OK,
> >> > > I will
> >> > > > > > > > > release
> >> > > > > > > > > > > > > > 3.0.0-beta-2,
> >> > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the
> >> > > 3.0.0-beta-2
> >> > > > > > tag
> >> > > > > > > > > and make the
> >> > > > > > > > > > > > > > 3.0.0
> >> > > > > > > > > > > > > > > > > > > > release soon.
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > Hope I can get all these things done
> >> > > within the
> >> > > > > > > > year
> >> > > > > > > > > 2024.
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > Thanks.
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
> >> > > > > > > > *Email*: stoty@cloudera.com
> >> > > > > > > > cloudera.com <https://www.cloudera.com>
> >> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> >> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> >> > > [image:
> >> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> >> > > [image:
> >> > > > > > Cloudera
> >> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >> > > > > > > > ------------------------------
> >> > > > > > > > ------------------------------
> >> > > > > > > >
> >> > > > > >
> >> > >
> >> >
> >> >
> >> > --
> >> > *István Tóth* | Sr. Staff Software Engineer
> >> > *Email*: stoty@cloudera.com
> >> > cloudera.com <https://www.cloudera.com>
> >> > [image: Cloudera] <https://www.cloudera.com/>
> >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> >> Cloudera
> >> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >> > ------------------------------
> >> > ------------------------------
> >>
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: stoty@cloudera.com
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> > ------------------------------
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------

Re: [DISCUSS] For 3.0.0-beta-2 and the final 3.0.0 release

[Istvan Toth <st...@cloudera.com>]

This seems to have been caused by forward synchronous RPC changes between
Hbase 2 and 3 (where HBase 2 simulates non-inclusive start keys by
adding a \00 byte and setting them as inclusive, while HBase 3 handles them
natively.
This is something that we can handle on the Phoenix side.

On the other hand, I've found a major and a minor breaking change in HBase
3:

The big problem is HBASE-29843
<https://issues.apache.org/jira/browse/HBASE-29843>
HBaseRPCController.setPriority(final TableName tn) is never called on
branch-3+
Phoenix relies on setting it its own HBaseRPCController to set custom
priorities, but HBase 3 has mostly abandoned using HBaseRPCController,
leaving no
working mechanism for Phoenix to use.
HBase needs to restore using it to maintain compatibility. (or provide some
alternative method)

The smaller problem is HBASE-29834
<https://issues.apache.org/jira/browse/HBASE-29834> MILLIS_BETWEEN_NEXTS
metric is not updated on branch-3+
This does not break Phoenix, it just happens to be detected by a Phoenix
test.

Istvan


On Mon, Nov 3, 2025 at 10:22 AM Istvan Toth <st...@cloudera.com> wrote:

> Sorry, I missed your reply eralier, Duo.
>
> I still haven't figured out the issue (though at least now I am now able
> to run those tests from a debugger)
> No ticket yet, as I'm still not sure which project to use.
>
> To repro this, we need to
> * install a current HBase 3 snapshot locally from branch-3 head,
> * check out my WIP Phoenix HBase 3 branch:
> https://github.com/stoty/phoenix/tree/PHOENIX-7164-2025
> * then run a faling test:
>
> *mvn clean verify -am -pl phoenix-core
> -Dit.test=BaseAggregateWithRegionMoves2IT#testGroupByArray
> -Dhbase.profile=3.0  -Dtest=foo*
> There are other similar tests failures, missing expected ResultSet rows
> after aggressively moving around the table regions during the query.
>
> Istvan
>
> On Thu, Oct 16, 2025 at 3:49 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> Is there an issue in HBase or Phoenix for tracking this problem?
>>
>> Thanks.
>>
>> Istvan Toth <st...@cloudera.com.invalid> 于2025年10月16日周四 11:58写道：
>> >
>> > Quick heads up:
>> > I'm investigating a regression in the Phoenix test suite where scans are
>> > not reset when the processed region is moved, leading to missing data
>> from
>> > the scan.
>> >
>> > I am not positive yet that the issue is on the HBase side, will report
>> here
>> > (and open a ticket if needed) when I know more.
>> >
>> > Istvan
>> >
>> > On Sat, Sep 13, 2025 at 2:18 PM 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>> >
>> > > Only 4 issues left on 3.0.0-beta-2 now, will wait for one more week to
>> > > see if we can get these issues done, otherwise will move them out of
>> > > 3.0.0.
>> > >
>> > > And the HBCK2 issue has been fixed.
>> > >
>> > > Plan to cut a new hbase-operator-tools release before cutting
>> 3.0.0-beta-2.
>> > >
>> > > Thanks.
>> > >
>> > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年8月31日周日 22:57写道：
>> > > >
>> > > > I've finished one round of reviewing pending issues for
>> 3.0.0-beta-2.
>> > > >
>> > > > There are still some necessary ones we need to get in before the
>> final
>> > > > 3.0.0 release, like make HBCK2 work against hbase3.
>> > > >
>> > > > Will keep working on them.
>> > > >
>> > > > Thanks.
>> > > >
>> > > > Nihal Jain <ni...@gmail.com> 于2025年6月19日周四 16:06写道：
>> > > > >
>> > > > > Yes, from hbase-thirdparty side, release is done and also
>> consumed in
>> > > hbase
>> > > > > branch-2.5 and branch-2.6 and we are good here.
>> > > > >
>> > > > > On Wed, 18 Jun 2025 at 8:48 PM, 张铎(Duo Zhang) <
>> palomino219@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Oh, I think we should cut 2.6.3 and 2.5.12 too? Since we have
>> already
>> > > > > > made the new thirdparty release without jetty 12.
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > > > Nihal Jain <ni...@gmail.com> 于2025年6月18日周三 01:51写道：
>> > > > > > >
>> > > > > > > Ah yes, let me restart work for hbase thirdparty release for
>> jetty
>> > > 12
>> > > > > > soon.
>> > > > > > >
>> > > > > > > On Tue, 17 Jun 2025 at 10:37 PM, Istvan Toth
>> > > <stoty@cloudera.com.invalid
>> > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > I think we should make sure we're using Jetty 12 in the next
>> > > beta. (
>> > > > > > so we
>> > > > > > > > need a thirdparty release for Jetty 12)
>> > > > > > > > We could also review other dependencies, and bump the ones
>> (if
>> > > any)
>> > > > > > that
>> > > > > > > > were kept on old versions because of the Java 8 requirement.
>> > > > > > > >
>> > > > > > > > Istvan
>> > > > > > > >
>> > > > > > > > On Tue, Jun 17, 2025 at 11:08 AM 张铎(Duo Zhang) <
>> > > palomino219@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > The good news is that I've finally got a successful 10B
>> ITBLL
>> > > run.
>> > > > > > > > >
>> > > > > > > > > You can check HBASE-27919 for more details.
>> > > > > > > > >
>> > > > > > > > > Next let me cleanup the jira issues and nightly runs.
>> > > > > > > > >
>> > > > > > > > > This time I think we can see 3.0.0 out soon.
>> > > > > > > > >
>> > > > > > > > > Thanks.
>> > > > > > > > >
>> > > > > > > > > Nihal Jain <ni...@apache.org> 于2025年3月12日周三 03:37写道：
>> > > > > > > > > >
>> > > > > > > > > > Hi Duo,
>> > > > > > > > > >
>> > > > > > > > > > Have you tried hbase-3.x on a kerberized env? I
>> encountered
>> > > > > > HBASE-29144
>> > > > > > > > > with kerberos. Posting here to just bring to attention.
>> > > > > > > > > >
>> > > > > > > > > > Regards
>> > > > > > > > > > Nihal
>> > > > > > > > > >
>> > > > > > > > > > On 2025/03/08 01:49:12 "张铎(Duo Zhang)" wrote:
>> > > > > > > > > > > A good news, cluster replication worked fine while
>> rolling
>> > > > > > upgrading
>> > > > > > > > > in my test.
>> > > > > > > > > > >
>> > > > > > > > > > > Let me focus on the region replication stuff. After
>> fixing
>> > > this
>> > > > > > > > issue,
>> > > > > > > > > > > the only blocker is a stable 10B ITBLL run.
>> > > > > > > > > > >
>> > > > > > > > > > > Hope we could get 3.0.0 out soon.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks.
>> > > > > > > > > > >
>> > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com> 于2025年3月6日周四
>> > > 22:55写道：
>> > > > > > > > > > > >
>> > > > > > > > > > > > I've finished one round of rolling upgrading test.
>> It
>> > > worked
>> > > > > > fine.
>> > > > > > > > > > > > Please see HBASE-29161.
>> > > > > > > > > > > >
>> > > > > > > > > > > > But soon I found a problem when memstore
>> replication is
>> > > > > > enabled,
>> > > > > > > > and
>> > > > > > > > > > > > filed HBASE-29169.
>> > > > > > > > > > > >
>> > > > > > > > > > > > I will test upgrading with cluster replication too,
>> to
>> > > see if
>> > > > > > there
>> > > > > > > > > > > > are any other issues.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Istvan Toth <st...@apache.org> 于2025年1月20日周一
>> 14:24写道：
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > For HBASE-28431, only the final step of adding the
>> > > > > > Hadoop-less
>> > > > > > > > > assembly is
>> > > > > > > > > > > > > missing.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Please review
>> > > https://github.com/apache/hbase/pull/5766 if
>> > > > > > you
>> > > > > > > > > have the
>> > > > > > > > > > > > > time.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Istvan
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Tue, Dec 31, 2024 at 2:43 PM 张铎(Duo Zhang) <
>> > > > > > > > > palomino219@gmail.com> wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > I've finished the first round of issue
>> alignment, it
>> > > took
>> > > > > > > > > several months...
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > And I've created a release audit tool for 3.0.0
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > https://github.com/apache9/hbase-release-audit
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > where we can have several ignore files, so we
>> do not
>> > > need
>> > > > > > to
>> > > > > > > > > manually
>> > > > > > > > > > > > > > check the result every time.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > So now, there are still several things to be
>> done
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > 1. Make sure we have all the features and fixes
>> we
>> > > want to
>> > > > > > land
>> > > > > > > > > on
>> > > > > > > > > > > > > > branch-3. For example, I think Istvan's assembly
>> > > cleanup
>> > > > > > should
>> > > > > > > > > be
>> > > > > > > > > > > > > > included. Please report other issues you want to
>> > > land in
>> > > > > > final
>> > > > > > > > > 3.0.0.
>> > > > > > > > > > > > > > 2. Several stable 10B ITBLL runs. I will try to
>> see
>> > > if I
>> > > > > > can
>> > > > > > > > get
>> > > > > > > > > > > > > > enough resources but it will be great if others
>> > > could help
>> > > > > > > > here.
>> > > > > > > > > > > > > > 3. Rolling upgrading test. I plan to do rolling
>> > > upgrade
>> > > > > > test
>> > > > > > > > > with the
>> > > > > > > > > > > > > > latest 2.6.x and 2.5.x releases. Since we have
>> big
>> > > changes
>> > > > > > on
>> > > > > > > > > > > > > > replication and region replica replication, I
>> plan
>> > > to have
>> > > > > > > > tests
>> > > > > > > > > with
>> > > > > > > > > > > > > > these two features enabled too if possible.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Any other suggestions?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thanks.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
>> 于2024年9月19日周四
>> > > > > > 22:35写道：
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > And do we want to apply the same trick to
>> > > MasterObserver
>> > > > > > and
>> > > > > > > > > > > > > > > RegionServerObserver?
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > For RegionObserver, we'd better do this
>> because
>> > > there are
>> > > > > > > > > already
>> > > > > > > > > > > > > > > methods in it that use the ObserverContext<?
>> > > extends
>> > > > > > > > > > > > > > > RegionCoprocessorEnvironment> declaration,
>> which
>> > > were
>> > > > > > added
>> > > > > > > > in
>> > > > > > > > > > > > > > > HBASE-10378.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > For other coprocessor observers, we do not
>> have
>> > > such
>> > > > > > methods
>> > > > > > > > > yet.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
>> 于2024年9月19日周四
>> > > > > > 22:03写道：
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > We do it here or we wait for 4.0. I'm
>> inclined
>> > > toward
>> > > > > > doing
>> > > > > > > > > it here
>> > > > > > > > > > > > > > > > because (i think) we expect coprocessor
>> > > > > > implementations to
>> > > > > > > > > necessarily
>> > > > > > > > > > > > > > > > recompile anyway. Though I admit I've not
>> thought
>> > > > > > through
>> > > > > > > > > what
>> > > > > > > > > > > > > > features are
>> > > > > > > > > > > > > > > > enabled by this new generic signature.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo
>> Zhang) <
>> > > > > > > > > palomino219@gmail.com>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Filed HBASE-28862.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > It will break old coprocessor
>> implementations
>> > > even if
>> > > > > > > > they
>> > > > > > > > > do not use
>> > > > > > > > > > > > > > > > > any deprecated method. But since it does
>> not
>> > > change
>> > > > > > any
>> > > > > > > > > abilities, I
>> > > > > > > > > > > > > > > > > think it is OK to include this in a major
>> > > release as
>> > > > > > we
>> > > > > > > > > have already
>> > > > > > > > > > > > > > > > > done bunch of breaking changes, like
>> changing
>> > > the
>> > > > > > > > protobuf
>> > > > > > > > > from
>> > > > > > > > > > > > > > > > > unshaded one to shaded and relocated
>> one...
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Thanks.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 张铎(Duo Zhang) <pa...@gmail.com>
>> > > 于2024年9月19日周四
>> > > > > > > > > 17:36写道：
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Thanks for pointing this TODO out.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Let me file an issue abort this. For me
>> I
>> > > think we
>> > > > > > > > > should do this
>> > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > 3.0.0.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Nick Dimiduk <nd...@apache.org>
>> > > 于2024年9月19日周四
>> > > > > > > > > 16:46写道：
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Should we take the opportunity to
>> upgrade
>> > > the
>> > > > > > generic
>> > > > > > > > > > > > > > definitions in
>> > > > > > > > > > > > > > > > > our
>> > > > > > > > > > > > > > > > > > > coprocessor signatures as well?
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > >
>> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > On Tue, Sep 17, 2024 at 4:41 PM Duo
>> Zhang <
>> > > > > > > > > zhangduo@apache.org>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > I've done several rounds of API
>> cleanup
>> > > in
>> > > > > > > > > HBASE-24888 and now
>> > > > > > > > > > > > > > for me
>> > > > > > > > > > > > > > > > > > > > I think it is enough for making the
>> new
>> > > major
>> > > > > > > > > release.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > I've filed HBASE-28844 for aligning
>> the
>> > > commit
>> > > > > > > > > history and jira
>> > > > > > > > > > > > > > > > > > > > issues, it will be a huge project
>> for a
>> > > major
>> > > > > > > > > release and may
>> > > > > > > > > > > > > > take
>> > > > > > > > > > > > > > > > > > > > several weeks.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > After that, I think we are good to
>> go.
>> > > And I
>> > > > > > will
>> > > > > > > > > also try to
>> > > > > > > > > > > > > > find
>> > > > > > > > > > > > > > > > > > > > some resources to run several
>> rounds of
>> > > ITBLL
>> > > > > > and
>> > > > > > > > > also some
>> > > > > > > > > > > > > > upgrading
>> > > > > > > > > > > > > > > > > > > > tests. Finally, after everything is
>> OK,
>> > > I will
>> > > > > > > > > release
>> > > > > > > > > > > > > > 3.0.0-beta-2,
>> > > > > > > > > > > > > > > > > > > > and then cut branch-3.0 from the
>> > > 3.0.0-beta-2
>> > > > > > tag
>> > > > > > > > > and make the
>> > > > > > > > > > > > > > 3.0.0
>> > > > > > > > > > > > > > > > > > > > release soon.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Hope I can get all these things done
>> > > within the
>> > > > > > > > year
>> > > > > > > > > 2024.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Thanks.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > *István Tóth* | Sr. Staff Software Engineer
>> > > > > > > > *Email*: stoty@cloudera.com
>> > > > > > > > cloudera.com <https://www.cloudera.com>
>> > > > > > > > [image: Cloudera] <https://www.cloudera.com/>
>> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
>> > > [image:
>> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
>> > > [image:
>> > > > > > Cloudera
>> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> > > > > > > > ------------------------------
>> > > > > > > > ------------------------------
>> > > > > > > >
>> > > > > >
>> > >
>> >
>> >
>> > --
>> > *István Tóth* | Sr. Staff Software Engineer
>> > *Email*: stoty@cloudera.com
>> > cloudera.com <https://www.cloudera.com>
>> > [image: Cloudera] <https://www.cloudera.com/>
>> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>> Cloudera
>> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> > ------------------------------
>> > ------------------------------
>>
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>


-- 
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------