You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Andor Molnar <an...@apache.org> on 2022/03/24 09:18:20 UTC
Meta replicas in LoadBalance mode broken in 2.4?
Hi HBase folks,
Does anybody actively use meta replicas in production and take advantage of the load balanced feature in 2.4?
We’ve verified it in a test cluster and the ITBLL test is dropping lots of errors.
Client side config:
hbase.locator.meta.replicas.mode = LoadBalance
hbase.meta.replicas.use = true
Server side config:
hbase.meta.replica.count = 3
Error in ITBLL generator run:
22/03/23 17:58:18 INFO mapreduce.Job: Task Id : attempt_1648045371733_0002_m_000002_2, Status : FAILED
Error: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 3120 actions: IntegrationTestBigLinkedList: 3120 times, servers with issues: null
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.makeException(BufferedMutatorImpl.java:333)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doFlush(BufferedMutatorImpl.java:307)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.close(BufferedMutatorImpl.java:235)
at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.cleanup(IntegrationTestBigLinkedList.java:524)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Can we say that the feature is broken or I’m missing something.
Thanks,
Andor
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Huaxiang Sun <hu...@apache.org>.
Hi Andor,
Please keep us updated how itbll goes with the new server side config. Your result will be used a datapoint as well.
Thanks,
Huaxiang
On 2022/03/25 17:51:49 Huaxiang Sun wrote:
> Hi Andor,
>
> I get what you are saying. The HFile refreshing is the old way for replica regions to refresh hfiles periodically, default is 5 minutes. In this itbll case, we need to have the wal replication enabled for meta replica. Please check out,
> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0. Basically, you need to set "hbase.region.replica.replication.catalog.enabled" to true in the configuration and rerun itbll. Otherwise, all meta changes at the primary meta region wont be updated at the replica meta regions and it will result in itbll failures.
>
> Hope this helps,
>
> Huaxiang
>
>
> On 2022/03/25 13:46:42 Andor Molnar wrote:
> > Hi Huaxiang,
> >
> > We use 2.4.6 for the tests.
> >
> > I run itbll with the following command:
> >
> > hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator 15 1000000 /tmp/hbase-itbll
> >
> > for the generator step and essentially jobs have failed. We can see the meta request are spanning out to replicas, but writes start failing after this due to the stale cache which is not getting updated.
> >
> > Would you please tell me more about ‘hfile refresh’ and how to configure it?
> >
> > Thanks,
> > Andor
> >
> >
> >
> >
> > > On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org> wrote:
> > >
> > > Hi Andor,
> > >
> > > Which 2.4 release do you test in your lab? We use this feature at production cluster with 2.4.5.
> > > At server side, we use hfile refresh instead of wal replication. I used to run itbll for each release with this feature enabled. How did you find the errors, did itbll fail?
> > >
> > > Regards,
> > > Huaxiang
> >
> >
>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Andor Molnar <an...@apache.org>.
Hi Huaxiang,
Thanks for the confirmation.
Andor
On Wed, 2022-04-06 at 21:59 +0000, Huaxiang Sun wrote:
> Sorry for replying late. Yeah, at server side, since region move is
> very rare in our production cluster, we did not enable the async wal
> replication. It is only being tested in itbll cluster.
>
> On 2022/03/31 13:35:01 Andor Molnar wrote:
> > One more question Huaxiang which makes me confused:
> >
> > You mentioned that "At server side, we use hfile refresh instead of
> > wal replication.” which means that async wal replication for meta
> > is not tested by you.
> >
> > Is that correct?
> >
> > Andor
> >
> >
> >
> >
> > > On 2022. Mar 30., at 19:22, Andor Molnar <an...@apache.org>
> > > wrote:
> > >
> > > Hi Huaxiang,
> > >
> > > Given that you already use this feature in production with no
> > > problems, I created a short patch to remove the “be careful”
> > > warnings from the HBase documentation.
> > >
> > > PTAL if you agree.
> > >
> > > https://github.com/apache/hbase/pull/4301
> > >
> > > Thanks,
> > > Andor
> > >
> > >
> > >
> > >
> > > > On 2022. Mar 29., at 18:52, Huaxiang Sun
> > > > <hu...@apache.org> wrote:
> > > >
> > > > This is great, thanks for the testing results!
> > > >
> > > > Huaxiang
> > > >
> > > > On 2022/03/29 13:29:48 Andor Molnar wrote:
> > > > > Works!
> > > > >
> > > > > I enabled async wal replication with the suggested option and
> > > > > ITBLL ran successfully.
> > > > >
> > > > > generator step:
> > > > > hbase
> > > > > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
> > > > > generator 15 1000000 /tmp/hbase-itbll
> > > > >
> > > > > verification step:
> > > > > hbase
> > > > > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
> > > > > verify /tmp/hbase-itbll-verify 15
> > > > >
> > > > > Both succeeded. I also confirmed that meta replicas are
> > > > > written and read by the clients, so must be in load balance
> > > > > mode.
> > > > >
> > > > > Thanks for the help!
> > > > >
> > > > > Andor
> > > > >
> > > > >
> > > > >
> > > > > > On 2022. Mar 27., at 6:59, Huaxiang Sun
> > > > > > <hu...@apache.org> wrote:
> > > > > >
> > > > > > It makes sense to turn on async wal replication when
> > > > > > hbase.meta.replicas.use = true. Let me run couple rounds of
> > > > > > itbll with hbase.region.replica.replication.catalog.enabled
> > > > > > (lastest 2.4 and 2.5.0 candidates) to get more confidence
> > > > > > before proposing turn on async wal replication for meta.
> > > > > >
> > > > > > Thanks,
> > > > > > Huaxiang
> > > > > >
> > > > > > On 2022/03/26 04:03:15 Andrew Purtell wrote:
> > > > > > > Just to be clear when I say "it seems pointless to have
> > > > > > > meta replicas which
> > > > > > > do not actually receive updates (by default)", what I
> > > > > > > should have said is
> > > > > > > 'timely updates', because a long delay in updating meta
> > > > > > > might as well be a
> > > > > > > missed update.
> > > > > > >
> > > > > > > On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell
> > > > > > > <ap...@apache.org> wrote:
> > > > > > >
> > > > > > > > > "Async WAL replication for META is added as a new
> > > > > > > > > feature in 2.4.0. It
> > > > > > > > is still under active development. Use with caution.
> > > > > > > > Set
> > > > > > > > hbase.region.replica.replication.catalog.enabled to
> > > > > > > > enable async WAL
> > > > > > > > Replication for META region replicas. It is off by
> > > > > > > > default."
> > > > > > > >
> > > > > > > > Do we still need this warning?
> > > > > > > >
> > > > > > > > Should hbase.region.replica.replication.catalog.enabled
> > > > > > > > have a default of
> > > > > > > > 'true' (enabled) if hbase.meta.replicas.use = true ?
> > > > > > > > Otherwise, it seems
> > > > > > > > pointless to have meta replicas which do not actually
> > > > > > > > receive updates (by
> > > > > > > > default).
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun
> > > > > > > > <hu...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Andor,
> > > > > > > > >
> > > > > > > > > I get what you are saying. The HFile refreshing is
> > > > > > > > > the old way for
> > > > > > > > > replica regions to refresh hfiles periodically,
> > > > > > > > > default is 5 minutes. In
> > > > > > > > > this itbll case, we need to have the wal replication
> > > > > > > > > enabled for meta
> > > > > > > > > replica. Please check out,
> > > > > > > > >
> > > > > > > > > https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0
> > > > > > > > > .
> > > > > > > > > Basically, you need to set
> > > > > > > > > "hbase.region.replica.replication.catalog.enabled" to
> > > > > > > > > true in the
> > > > > > > > > configuration and rerun itbll. Otherwise, all meta
> > > > > > > > > changes at the primary
> > > > > > > > > meta region wont be updated at the replica meta
> > > > > > > > > regions and it will result
> > > > > > > > > in itbll failures.
> > > > > > > > >
> > > > > > > > > Hope this helps,
> > > > > > > > >
> > > > > > > > > Huaxiang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 2022/03/25 13:46:42 Andor Molnar wrote:
> > > > > > > > > > Hi Huaxiang,
> > > > > > > > > >
> > > > > > > > > > We use 2.4.6 for the tests.
> > > > > > > > > >
> > > > > > > > > > I run itbll with the following command:
> > > > > > > > > >
> > > > > > > > > > hbase
> > > > > > > > > > org.apache.hadoop.hbase.test.IntegrationTestBigLink
> > > > > > > > > > edList
> > > > > > > > > generator 15 1000000 /tmp/hbase-itbll
> > > > > > > > > >
> > > > > > > > > > for the generator step and essentially jobs have
> > > > > > > > > > failed. We can see the
> > > > > > > > > meta request are spanning out to replicas, but writes
> > > > > > > > > start failing after
> > > > > > > > > this due to the stale cache which is not getting
> > > > > > > > > updated.
> > > > > > > > > >
> > > > > > > > > > Would you please tell me more about ‘hfile refresh’
> > > > > > > > > > and how to
> > > > > > > > > configure it?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Andor
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On 2022. Mar 24., at 17:43, Huaxiang Sun
> > > > > > > > > > > <hu...@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Andor,
> > > > > > > > > > >
> > > > > > > > > > > Which 2.4 release do you test in your lab? We use
> > > > > > > > > > > this feature at
> > > > > > > > > production cluster with 2.4.5.
> > > > > > > > > > > At server side, we use hfile refresh instead of
> > > > > > > > > > > wal replication. I
> > > > > > > > > used to run itbll for each release with this feature
> > > > > > > > > enabled. How did you
> > > > > > > > > find the errors, did itbll fail?
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Huaxiang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best regards,
> > > > > > > > Andrew
> > > > > > > >
> > > > > > > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > > > > > > It's what we’ve earned
> > > > > > > > Welcome, apocalypse, what’s taken you so long?
> > > > > > > > Bring us the fitting end that we’ve been counting on
> > > > > > > > - A23, Welcome, Apocalypse
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Andrew
> > > > > > >
> > > > > > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > > > > > It's what we’ve earned
> > > > > > > Welcome, apocalypse, what’s taken you so long?
> > > > > > > Bring us the fitting end that we’ve been counting on
> > > > > > > - A23, Welcome, Apocalypse
> > > > > > >
> > > > >
> > > > >
> > >
> >
> >
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Huaxiang Sun <hu...@apache.org>.
Sorry for replying late. Yeah, at server side, since region move is very rare in our production cluster, we did not enable the async wal replication. It is only being tested in itbll cluster.
On 2022/03/31 13:35:01 Andor Molnar wrote:
> One more question Huaxiang which makes me confused:
>
> You mentioned that "At server side, we use hfile refresh instead of wal replication.” which means that async wal replication for meta is not tested by you.
>
> Is that correct?
>
> Andor
>
>
>
>
> > On 2022. Mar 30., at 19:22, Andor Molnar <an...@apache.org> wrote:
> >
> > Hi Huaxiang,
> >
> > Given that you already use this feature in production with no problems, I created a short patch to remove the “be careful” warnings from the HBase documentation.
> >
> > PTAL if you agree.
> >
> > https://github.com/apache/hbase/pull/4301
> >
> > Thanks,
> > Andor
> >
> >
> >
> >
> >> On 2022. Mar 29., at 18:52, Huaxiang Sun <hu...@apache.org> wrote:
> >>
> >> This is great, thanks for the testing results!
> >>
> >> Huaxiang
> >>
> >> On 2022/03/29 13:29:48 Andor Molnar wrote:
> >>> Works!
> >>>
> >>> I enabled async wal replication with the suggested option and ITBLL ran successfully.
> >>>
> >>> generator step:
> >>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator 15 1000000 /tmp/hbase-itbll
> >>>
> >>> verification step:
> >>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList verify /tmp/hbase-itbll-verify 15
> >>>
> >>> Both succeeded. I also confirmed that meta replicas are written and read by the clients, so must be in load balance mode.
> >>>
> >>> Thanks for the help!
> >>>
> >>> Andor
> >>>
> >>>
> >>>
> >>>> On 2022. Mar 27., at 6:59, Huaxiang Sun <hu...@apache.org> wrote:
> >>>>
> >>>> It makes sense to turn on async wal replication when hbase.meta.replicas.use = true. Let me run couple rounds of itbll with hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 candidates) to get more confidence before proposing turn on async wal replication for meta.
> >>>>
> >>>> Thanks,
> >>>> Huaxiang
> >>>>
> >>>> On 2022/03/26 04:03:15 Andrew Purtell wrote:
> >>>>> Just to be clear when I say "it seems pointless to have meta replicas which
> >>>>> do not actually receive updates (by default)", what I should have said is
> >>>>> 'timely updates', because a long delay in updating meta might as well be a
> >>>>> missed update.
> >>>>>
> >>>>> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org> wrote:
> >>>>>
> >>>>>>> "Async WAL replication for META is added as a new feature in 2.4.0. It
> >>>>>> is still under active development. Use with caution. Set
> >>>>>> hbase.region.replica.replication.catalog.enabled to enable async WAL
> >>>>>> Replication for META region replicas. It is off by default."
> >>>>>>
> >>>>>> Do we still need this warning?
> >>>>>>
> >>>>>> Should hbase.region.replica.replication.catalog.enabled have a default of
> >>>>>> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems
> >>>>>> pointless to have meta replicas which do not actually receive updates (by
> >>>>>> default).
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Andor,
> >>>>>>>
> >>>>>>> I get what you are saying. The HFile refreshing is the old way for
> >>>>>>> replica regions to refresh hfiles periodically, default is 5 minutes. In
> >>>>>>> this itbll case, we need to have the wal replication enabled for meta
> >>>>>>> replica. Please check out,
> >>>>>>>
> >>>>>>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0.
> >>>>>>> Basically, you need to set
> >>>>>>> "hbase.region.replica.replication.catalog.enabled" to true in the
> >>>>>>> configuration and rerun itbll. Otherwise, all meta changes at the primary
> >>>>>>> meta region wont be updated at the replica meta regions and it will result
> >>>>>>> in itbll failures.
> >>>>>>>
> >>>>>>> Hope this helps,
> >>>>>>>
> >>>>>>> Huaxiang
> >>>>>>>
> >>>>>>>
> >>>>>>> On 2022/03/25 13:46:42 Andor Molnar wrote:
> >>>>>>>> Hi Huaxiang,
> >>>>>>>>
> >>>>>>>> We use 2.4.6 for the tests.
> >>>>>>>>
> >>>>>>>> I run itbll with the following command:
> >>>>>>>>
> >>>>>>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
> >>>>>>> generator 15 1000000 /tmp/hbase-itbll
> >>>>>>>>
> >>>>>>>> for the generator step and essentially jobs have failed. We can see the
> >>>>>>> meta request are spanning out to replicas, but writes start failing after
> >>>>>>> this due to the stale cache which is not getting updated.
> >>>>>>>>
> >>>>>>>> Would you please tell me more about ‘hfile refresh’ and how to
> >>>>>>> configure it?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Andor
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Andor,
> >>>>>>>>>
> >>>>>>>>> Which 2.4 release do you test in your lab? We use this feature at
> >>>>>>> production cluster with 2.4.5.
> >>>>>>>>> At server side, we use hfile refresh instead of wal replication. I
> >>>>>>> used to run itbll for each release with this feature enabled. How did you
> >>>>>>> find the errors, did itbll fail?
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Huaxiang
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best regards,
> >>>>>> Andrew
> >>>>>>
> >>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
> >>>>>> It's what we’ve earned
> >>>>>> Welcome, apocalypse, what’s taken you so long?
> >>>>>> Bring us the fitting end that we’ve been counting on
> >>>>>> - A23, Welcome, Apocalypse
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Best regards,
> >>>>> Andrew
> >>>>>
> >>>>> Unrest, ignorance distilled, nihilistic imbeciles -
> >>>>> It's what we’ve earned
> >>>>> Welcome, apocalypse, what’s taken you so long?
> >>>>> Bring us the fitting end that we’ve been counting on
> >>>>> - A23, Welcome, Apocalypse
> >>>>>
> >>>
> >>>
> >
>
>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Andor Molnar <an...@apache.org>.
One more question Huaxiang which makes me confused:
You mentioned that "At server side, we use hfile refresh instead of wal replication.” which means that async wal replication for meta is not tested by you.
Is that correct?
Andor
> On 2022. Mar 30., at 19:22, Andor Molnar <an...@apache.org> wrote:
>
> Hi Huaxiang,
>
> Given that you already use this feature in production with no problems, I created a short patch to remove the “be careful” warnings from the HBase documentation.
>
> PTAL if you agree.
>
> https://github.com/apache/hbase/pull/4301
>
> Thanks,
> Andor
>
>
>
>
>> On 2022. Mar 29., at 18:52, Huaxiang Sun <hu...@apache.org> wrote:
>>
>> This is great, thanks for the testing results!
>>
>> Huaxiang
>>
>> On 2022/03/29 13:29:48 Andor Molnar wrote:
>>> Works!
>>>
>>> I enabled async wal replication with the suggested option and ITBLL ran successfully.
>>>
>>> generator step:
>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator 15 1000000 /tmp/hbase-itbll
>>>
>>> verification step:
>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList verify /tmp/hbase-itbll-verify 15
>>>
>>> Both succeeded. I also confirmed that meta replicas are written and read by the clients, so must be in load balance mode.
>>>
>>> Thanks for the help!
>>>
>>> Andor
>>>
>>>
>>>
>>>> On 2022. Mar 27., at 6:59, Huaxiang Sun <hu...@apache.org> wrote:
>>>>
>>>> It makes sense to turn on async wal replication when hbase.meta.replicas.use = true. Let me run couple rounds of itbll with hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 candidates) to get more confidence before proposing turn on async wal replication for meta.
>>>>
>>>> Thanks,
>>>> Huaxiang
>>>>
>>>> On 2022/03/26 04:03:15 Andrew Purtell wrote:
>>>>> Just to be clear when I say "it seems pointless to have meta replicas which
>>>>> do not actually receive updates (by default)", what I should have said is
>>>>> 'timely updates', because a long delay in updating meta might as well be a
>>>>> missed update.
>>>>>
>>>>> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org> wrote:
>>>>>
>>>>>>> "Async WAL replication for META is added as a new feature in 2.4.0. It
>>>>>> is still under active development. Use with caution. Set
>>>>>> hbase.region.replica.replication.catalog.enabled to enable async WAL
>>>>>> Replication for META region replicas. It is off by default."
>>>>>>
>>>>>> Do we still need this warning?
>>>>>>
>>>>>> Should hbase.region.replica.replication.catalog.enabled have a default of
>>>>>> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems
>>>>>> pointless to have meta replicas which do not actually receive updates (by
>>>>>> default).
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Andor,
>>>>>>>
>>>>>>> I get what you are saying. The HFile refreshing is the old way for
>>>>>>> replica regions to refresh hfiles periodically, default is 5 minutes. In
>>>>>>> this itbll case, we need to have the wal replication enabled for meta
>>>>>>> replica. Please check out,
>>>>>>>
>>>>>>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0.
>>>>>>> Basically, you need to set
>>>>>>> "hbase.region.replica.replication.catalog.enabled" to true in the
>>>>>>> configuration and rerun itbll. Otherwise, all meta changes at the primary
>>>>>>> meta region wont be updated at the replica meta regions and it will result
>>>>>>> in itbll failures.
>>>>>>>
>>>>>>> Hope this helps,
>>>>>>>
>>>>>>> Huaxiang
>>>>>>>
>>>>>>>
>>>>>>> On 2022/03/25 13:46:42 Andor Molnar wrote:
>>>>>>>> Hi Huaxiang,
>>>>>>>>
>>>>>>>> We use 2.4.6 for the tests.
>>>>>>>>
>>>>>>>> I run itbll with the following command:
>>>>>>>>
>>>>>>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
>>>>>>> generator 15 1000000 /tmp/hbase-itbll
>>>>>>>>
>>>>>>>> for the generator step and essentially jobs have failed. We can see the
>>>>>>> meta request are spanning out to replicas, but writes start failing after
>>>>>>> this due to the stale cache which is not getting updated.
>>>>>>>>
>>>>>>>> Would you please tell me more about ‘hfile refresh’ and how to
>>>>>>> configure it?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andor
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Andor,
>>>>>>>>>
>>>>>>>>> Which 2.4 release do you test in your lab? We use this feature at
>>>>>>> production cluster with 2.4.5.
>>>>>>>>> At server side, we use hfile refresh instead of wal replication. I
>>>>>>> used to run itbll for each release with this feature enabled. How did you
>>>>>>> find the errors, did itbll fail?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Huaxiang
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Andrew
>>>>>>
>>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>>>>> It's what we’ve earned
>>>>>> Welcome, apocalypse, what’s taken you so long?
>>>>>> Bring us the fitting end that we’ve been counting on
>>>>>> - A23, Welcome, Apocalypse
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Andrew
>>>>>
>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>>>> It's what we’ve earned
>>>>> Welcome, apocalypse, what’s taken you so long?
>>>>> Bring us the fitting end that we’ve been counting on
>>>>> - A23, Welcome, Apocalypse
>>>>>
>>>
>>>
>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Andor Molnar <an...@apache.org>.
Hi Huaxiang,
Given that you already use this feature in production with no problems, I created a short patch to remove the “be careful” warnings from the HBase documentation.
PTAL if you agree.
https://github.com/apache/hbase/pull/4301
Thanks,
Andor
> On 2022. Mar 29., at 18:52, Huaxiang Sun <hu...@apache.org> wrote:
>
> This is great, thanks for the testing results!
>
> Huaxiang
>
> On 2022/03/29 13:29:48 Andor Molnar wrote:
>> Works!
>>
>> I enabled async wal replication with the suggested option and ITBLL ran successfully.
>>
>> generator step:
>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator 15 1000000 /tmp/hbase-itbll
>>
>> verification step:
>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList verify /tmp/hbase-itbll-verify 15
>>
>> Both succeeded. I also confirmed that meta replicas are written and read by the clients, so must be in load balance mode.
>>
>> Thanks for the help!
>>
>> Andor
>>
>>
>>
>>> On 2022. Mar 27., at 6:59, Huaxiang Sun <hu...@apache.org> wrote:
>>>
>>> It makes sense to turn on async wal replication when hbase.meta.replicas.use = true. Let me run couple rounds of itbll with hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 candidates) to get more confidence before proposing turn on async wal replication for meta.
>>>
>>> Thanks,
>>> Huaxiang
>>>
>>> On 2022/03/26 04:03:15 Andrew Purtell wrote:
>>>> Just to be clear when I say "it seems pointless to have meta replicas which
>>>> do not actually receive updates (by default)", what I should have said is
>>>> 'timely updates', because a long delay in updating meta might as well be a
>>>> missed update.
>>>>
>>>> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org> wrote:
>>>>
>>>>>> "Async WAL replication for META is added as a new feature in 2.4.0. It
>>>>> is still under active development. Use with caution. Set
>>>>> hbase.region.replica.replication.catalog.enabled to enable async WAL
>>>>> Replication for META region replicas. It is off by default."
>>>>>
>>>>> Do we still need this warning?
>>>>>
>>>>> Should hbase.region.replica.replication.catalog.enabled have a default of
>>>>> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems
>>>>> pointless to have meta replicas which do not actually receive updates (by
>>>>> default).
>>>>>
>>>>>
>>>>> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Andor,
>>>>>>
>>>>>> I get what you are saying. The HFile refreshing is the old way for
>>>>>> replica regions to refresh hfiles periodically, default is 5 minutes. In
>>>>>> this itbll case, we need to have the wal replication enabled for meta
>>>>>> replica. Please check out,
>>>>>>
>>>>>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0.
>>>>>> Basically, you need to set
>>>>>> "hbase.region.replica.replication.catalog.enabled" to true in the
>>>>>> configuration and rerun itbll. Otherwise, all meta changes at the primary
>>>>>> meta region wont be updated at the replica meta regions and it will result
>>>>>> in itbll failures.
>>>>>>
>>>>>> Hope this helps,
>>>>>>
>>>>>> Huaxiang
>>>>>>
>>>>>>
>>>>>> On 2022/03/25 13:46:42 Andor Molnar wrote:
>>>>>>> Hi Huaxiang,
>>>>>>>
>>>>>>> We use 2.4.6 for the tests.
>>>>>>>
>>>>>>> I run itbll with the following command:
>>>>>>>
>>>>>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
>>>>>> generator 15 1000000 /tmp/hbase-itbll
>>>>>>>
>>>>>>> for the generator step and essentially jobs have failed. We can see the
>>>>>> meta request are spanning out to replicas, but writes start failing after
>>>>>> this due to the stale cache which is not getting updated.
>>>>>>>
>>>>>>> Would you please tell me more about ‘hfile refresh’ and how to
>>>>>> configure it?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andor
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Andor,
>>>>>>>>
>>>>>>>> Which 2.4 release do you test in your lab? We use this feature at
>>>>>> production cluster with 2.4.5.
>>>>>>>> At server side, we use hfile refresh instead of wal replication. I
>>>>>> used to run itbll for each release with this feature enabled. How did you
>>>>>> find the errors, did itbll fail?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Huaxiang
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Andrew
>>>>>
>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>>>> It's what we’ve earned
>>>>> Welcome, apocalypse, what’s taken you so long?
>>>>> Bring us the fitting end that we’ve been counting on
>>>>> - A23, Welcome, Apocalypse
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrew
>>>>
>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>>> It's what we’ve earned
>>>> Welcome, apocalypse, what’s taken you so long?
>>>> Bring us the fitting end that we’ve been counting on
>>>> - A23, Welcome, Apocalypse
>>>>
>>
>>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Huaxiang Sun <hu...@apache.org>.
This is great, thanks for the testing results!
Huaxiang
On 2022/03/29 13:29:48 Andor Molnar wrote:
> Works!
>
> I enabled async wal replication with the suggested option and ITBLL ran successfully.
>
> generator step:
> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator 15 1000000 /tmp/hbase-itbll
>
> verification step:
> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList verify /tmp/hbase-itbll-verify 15
>
> Both succeeded. I also confirmed that meta replicas are written and read by the clients, so must be in load balance mode.
>
> Thanks for the help!
>
> Andor
>
>
>
> > On 2022. Mar 27., at 6:59, Huaxiang Sun <hu...@apache.org> wrote:
> >
> > It makes sense to turn on async wal replication when hbase.meta.replicas.use = true. Let me run couple rounds of itbll with hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 candidates) to get more confidence before proposing turn on async wal replication for meta.
> >
> > Thanks,
> > Huaxiang
> >
> > On 2022/03/26 04:03:15 Andrew Purtell wrote:
> >> Just to be clear when I say "it seems pointless to have meta replicas which
> >> do not actually receive updates (by default)", what I should have said is
> >> 'timely updates', because a long delay in updating meta might as well be a
> >> missed update.
> >>
> >> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org> wrote:
> >>
> >>>> "Async WAL replication for META is added as a new feature in 2.4.0. It
> >>> is still under active development. Use with caution. Set
> >>> hbase.region.replica.replication.catalog.enabled to enable async WAL
> >>> Replication for META region replicas. It is off by default."
> >>>
> >>> Do we still need this warning?
> >>>
> >>> Should hbase.region.replica.replication.catalog.enabled have a default of
> >>> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems
> >>> pointless to have meta replicas which do not actually receive updates (by
> >>> default).
> >>>
> >>>
> >>> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
> >>> wrote:
> >>>
> >>>> Hi Andor,
> >>>>
> >>>> I get what you are saying. The HFile refreshing is the old way for
> >>>> replica regions to refresh hfiles periodically, default is 5 minutes. In
> >>>> this itbll case, we need to have the wal replication enabled for meta
> >>>> replica. Please check out,
> >>>>
> >>>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0.
> >>>> Basically, you need to set
> >>>> "hbase.region.replica.replication.catalog.enabled" to true in the
> >>>> configuration and rerun itbll. Otherwise, all meta changes at the primary
> >>>> meta region wont be updated at the replica meta regions and it will result
> >>>> in itbll failures.
> >>>>
> >>>> Hope this helps,
> >>>>
> >>>> Huaxiang
> >>>>
> >>>>
> >>>> On 2022/03/25 13:46:42 Andor Molnar wrote:
> >>>>> Hi Huaxiang,
> >>>>>
> >>>>> We use 2.4.6 for the tests.
> >>>>>
> >>>>> I run itbll with the following command:
> >>>>>
> >>>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
> >>>> generator 15 1000000 /tmp/hbase-itbll
> >>>>>
> >>>>> for the generator step and essentially jobs have failed. We can see the
> >>>> meta request are spanning out to replicas, but writes start failing after
> >>>> this due to the stale cache which is not getting updated.
> >>>>>
> >>>>> Would you please tell me more about ‘hfile refresh’ and how to
> >>>> configure it?
> >>>>>
> >>>>> Thanks,
> >>>>> Andor
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
> >>>> wrote:
> >>>>>>
> >>>>>> Hi Andor,
> >>>>>>
> >>>>>> Which 2.4 release do you test in your lab? We use this feature at
> >>>> production cluster with 2.4.5.
> >>>>>> At server side, we use hfile refresh instead of wal replication. I
> >>>> used to run itbll for each release with this feature enabled. How did you
> >>>> find the errors, did itbll fail?
> >>>>>>
> >>>>>> Regards,
> >>>>>> Huaxiang
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Best regards,
> >>> Andrew
> >>>
> >>> Unrest, ignorance distilled, nihilistic imbeciles -
> >>> It's what we’ve earned
> >>> Welcome, apocalypse, what’s taken you so long?
> >>> Bring us the fitting end that we’ve been counting on
> >>> - A23, Welcome, Apocalypse
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrew
> >>
> >> Unrest, ignorance distilled, nihilistic imbeciles -
> >> It's what we’ve earned
> >> Welcome, apocalypse, what’s taken you so long?
> >> Bring us the fitting end that we’ve been counting on
> >> - A23, Welcome, Apocalypse
> >>
>
>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Andor Molnar <an...@apache.org>.
Works!
I enabled async wal replication with the suggested option and ITBLL ran successfully.
generator step:
hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator 15 1000000 /tmp/hbase-itbll
verification step:
hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList verify /tmp/hbase-itbll-verify 15
Both succeeded. I also confirmed that meta replicas are written and read by the clients, so must be in load balance mode.
Thanks for the help!
Andor
> On 2022. Mar 27., at 6:59, Huaxiang Sun <hu...@apache.org> wrote:
>
> It makes sense to turn on async wal replication when hbase.meta.replicas.use = true. Let me run couple rounds of itbll with hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 candidates) to get more confidence before proposing turn on async wal replication for meta.
>
> Thanks,
> Huaxiang
>
> On 2022/03/26 04:03:15 Andrew Purtell wrote:
>> Just to be clear when I say "it seems pointless to have meta replicas which
>> do not actually receive updates (by default)", what I should have said is
>> 'timely updates', because a long delay in updating meta might as well be a
>> missed update.
>>
>> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org> wrote:
>>
>>>> "Async WAL replication for META is added as a new feature in 2.4.0. It
>>> is still under active development. Use with caution. Set
>>> hbase.region.replica.replication.catalog.enabled to enable async WAL
>>> Replication for META region replicas. It is off by default."
>>>
>>> Do we still need this warning?
>>>
>>> Should hbase.region.replica.replication.catalog.enabled have a default of
>>> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems
>>> pointless to have meta replicas which do not actually receive updates (by
>>> default).
>>>
>>>
>>> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
>>> wrote:
>>>
>>>> Hi Andor,
>>>>
>>>> I get what you are saying. The HFile refreshing is the old way for
>>>> replica regions to refresh hfiles periodically, default is 5 minutes. In
>>>> this itbll case, we need to have the wal replication enabled for meta
>>>> replica. Please check out,
>>>>
>>>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0.
>>>> Basically, you need to set
>>>> "hbase.region.replica.replication.catalog.enabled" to true in the
>>>> configuration and rerun itbll. Otherwise, all meta changes at the primary
>>>> meta region wont be updated at the replica meta regions and it will result
>>>> in itbll failures.
>>>>
>>>> Hope this helps,
>>>>
>>>> Huaxiang
>>>>
>>>>
>>>> On 2022/03/25 13:46:42 Andor Molnar wrote:
>>>>> Hi Huaxiang,
>>>>>
>>>>> We use 2.4.6 for the tests.
>>>>>
>>>>> I run itbll with the following command:
>>>>>
>>>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
>>>> generator 15 1000000 /tmp/hbase-itbll
>>>>>
>>>>> for the generator step and essentially jobs have failed. We can see the
>>>> meta request are spanning out to replicas, but writes start failing after
>>>> this due to the stale cache which is not getting updated.
>>>>>
>>>>> Would you please tell me more about ‘hfile refresh’ and how to
>>>> configure it?
>>>>>
>>>>> Thanks,
>>>>> Andor
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
>>>> wrote:
>>>>>>
>>>>>> Hi Andor,
>>>>>>
>>>>>> Which 2.4 release do you test in your lab? We use this feature at
>>>> production cluster with 2.4.5.
>>>>>> At server side, we use hfile refresh instead of wal replication. I
>>>> used to run itbll for each release with this feature enabled. How did you
>>>> find the errors, did itbll fail?
>>>>>>
>>>>>> Regards,
>>>>>> Huaxiang
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrew
>>>
>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>> It's what we’ve earned
>>> Welcome, apocalypse, what’s taken you so long?
>>> Bring us the fitting end that we’ve been counting on
>>> - A23, Welcome, Apocalypse
>>>
>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Unrest, ignorance distilled, nihilistic imbeciles -
>> It's what we’ve earned
>> Welcome, apocalypse, what’s taken you so long?
>> Bring us the fitting end that we’ve been counting on
>> - A23, Welcome, Apocalypse
>>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Andor Molnar <an...@apache.org>.
Thanks a lot Huaxiang!
Sorry I didn’t have a chance to follow-up on this today, but hopefully come back to you tomorrow.
Really appreciate your efforts.
Andor
> On 2022. Mar 28., at 19:10, Huaxiang Sun <hu...@apache.org> wrote:
>
> I will start to work on it.
>
> Thanks
> Huaxiang
>
> On 2022/03/27 18:42:06 Andrew Purtell wrote:
>> On Sat, Mar 26, 2022 at 10:00 PM Huaxiang Sun <hu...@apache.org>
>> wrote:
>>
>>> It makes sense to turn on async wal replication when
>>> hbase.meta.replicas.use = true. Let me run couple rounds of itbll with
>>> hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0
>>> candidates) to get more confidence before proposing turn on async wal
>>> replication for meta.
>>>
>>
>> +1
>>
>> Thanks,
>>> Huaxiang
>>>
>>> On 2022/03/26 04:03:15 Andrew Purtell wrote:
>>>> Just to be clear when I say "it seems pointless to have meta replicas
>>> which
>>>> do not actually receive updates (by default)", what I should have said is
>>>> 'timely updates', because a long delay in updating meta might as well be
>>> a
>>>> missed update.
>>>>
>>>> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org>
>>> wrote:
>>>>
>>>>>> "Async WAL replication for META is added as a new feature in 2.4.0.
>>> It
>>>>> is still under active development. Use with caution. Set
>>>>> hbase.region.replica.replication.catalog.enabled to enable async WAL
>>>>> Replication for META region replicas. It is off by default."
>>>>>
>>>>> Do we still need this warning?
>>>>>
>>>>> Should hbase.region.replica.replication.catalog.enabled have a default
>>> of
>>>>> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it
>>> seems
>>>>> pointless to have meta replicas which do not actually receive updates
>>> (by
>>>>> default).
>>>>>
>>>>>
>>>>> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Andor,
>>>>>>
>>>>>> I get what you are saying. The HFile refreshing is the old way for
>>>>>> replica regions to refresh hfiles periodically, default is 5 minutes.
>>> In
>>>>>> this itbll case, we need to have the wal replication enabled for meta
>>>>>> replica. Please check out,
>>>>>>
>>>>>>
>>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0
>>> .
>>>>>> Basically, you need to set
>>>>>> "hbase.region.replica.replication.catalog.enabled" to true in the
>>>>>> configuration and rerun itbll. Otherwise, all meta changes at the
>>> primary
>>>>>> meta region wont be updated at the replica meta regions and it will
>>> result
>>>>>> in itbll failures.
>>>>>>
>>>>>> Hope this helps,
>>>>>>
>>>>>> Huaxiang
>>>>>>
>>>>>>
>>>>>> On 2022/03/25 13:46:42 Andor Molnar wrote:
>>>>>>> Hi Huaxiang,
>>>>>>>
>>>>>>> We use 2.4.6 for the tests.
>>>>>>>
>>>>>>> I run itbll with the following command:
>>>>>>>
>>>>>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
>>>>>> generator 15 1000000 /tmp/hbase-itbll
>>>>>>>
>>>>>>> for the generator step and essentially jobs have failed. We can see
>>> the
>>>>>> meta request are spanning out to replicas, but writes start failing
>>> after
>>>>>> this due to the stale cache which is not getting updated.
>>>>>>>
>>>>>>> Would you please tell me more about ‘hfile refresh’ and how to
>>>>>> configure it?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andor
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Andor,
>>>>>>>>
>>>>>>>> Which 2.4 release do you test in your lab? We use this feature
>>> at
>>>>>> production cluster with 2.4.5.
>>>>>>>> At server side, we use hfile refresh instead of wal replication. I
>>>>>> used to run itbll for each release with this feature enabled. How did
>>> you
>>>>>> find the errors, did itbll fail?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Huaxiang
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Andrew
>>>>>
>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>>>> It's what we’ve earned
>>>>> Welcome, apocalypse, what’s taken you so long?
>>>>> Bring us the fitting end that we’ve been counting on
>>>>> - A23, Welcome, Apocalypse
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrew
>>>>
>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>>> It's what we’ve earned
>>>> Welcome, apocalypse, what’s taken you so long?
>>>> Bring us the fitting end that we’ve been counting on
>>>> - A23, Welcome, Apocalypse
>>>>
>>>
>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Unrest, ignorance distilled, nihilistic imbeciles -
>> It's what we’ve earned
>> Welcome, apocalypse, what’s taken you so long?
>> Bring us the fitting end that we’ve been counting on
>> - A23, Welcome, Apocalypse
>>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Huaxiang Sun <hu...@apache.org>.
I will start to work on it.
Thanks
Huaxiang
On 2022/03/27 18:42:06 Andrew Purtell wrote:
> On Sat, Mar 26, 2022 at 10:00 PM Huaxiang Sun <hu...@apache.org>
> wrote:
>
> > It makes sense to turn on async wal replication when
> > hbase.meta.replicas.use = true. Let me run couple rounds of itbll with
> > hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0
> > candidates) to get more confidence before proposing turn on async wal
> > replication for meta.
> >
>
> +1
>
> Thanks,
> > Huaxiang
> >
> > On 2022/03/26 04:03:15 Andrew Purtell wrote:
> > > Just to be clear when I say "it seems pointless to have meta replicas
> > which
> > > do not actually receive updates (by default)", what I should have said is
> > > 'timely updates', because a long delay in updating meta might as well be
> > a
> > > missed update.
> > >
> > > On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org>
> > wrote:
> > >
> > > > > "Async WAL replication for META is added as a new feature in 2.4.0.
> > It
> > > > is still under active development. Use with caution. Set
> > > > hbase.region.replica.replication.catalog.enabled to enable async WAL
> > > > Replication for META region replicas. It is off by default."
> > > >
> > > > Do we still need this warning?
> > > >
> > > > Should hbase.region.replica.replication.catalog.enabled have a default
> > of
> > > > 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it
> > seems
> > > > pointless to have meta replicas which do not actually receive updates
> > (by
> > > > default).
> > > >
> > > >
> > > > On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
> > > > wrote:
> > > >
> > > >> Hi Andor,
> > > >>
> > > >> I get what you are saying. The HFile refreshing is the old way for
> > > >> replica regions to refresh hfiles periodically, default is 5 minutes.
> > In
> > > >> this itbll case, we need to have the wal replication enabled for meta
> > > >> replica. Please check out,
> > > >>
> > > >>
> > https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0
> > .
> > > >> Basically, you need to set
> > > >> "hbase.region.replica.replication.catalog.enabled" to true in the
> > > >> configuration and rerun itbll. Otherwise, all meta changes at the
> > primary
> > > >> meta region wont be updated at the replica meta regions and it will
> > result
> > > >> in itbll failures.
> > > >>
> > > >> Hope this helps,
> > > >>
> > > >> Huaxiang
> > > >>
> > > >>
> > > >> On 2022/03/25 13:46:42 Andor Molnar wrote:
> > > >> > Hi Huaxiang,
> > > >> >
> > > >> > We use 2.4.6 for the tests.
> > > >> >
> > > >> > I run itbll with the following command:
> > > >> >
> > > >> > hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
> > > >> generator 15 1000000 /tmp/hbase-itbll
> > > >> >
> > > >> > for the generator step and essentially jobs have failed. We can see
> > the
> > > >> meta request are spanning out to replicas, but writes start failing
> > after
> > > >> this due to the stale cache which is not getting updated.
> > > >> >
> > > >> > Would you please tell me more about ‘hfile refresh’ and how to
> > > >> configure it?
> > > >> >
> > > >> > Thanks,
> > > >> > Andor
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > > On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
> > > >> wrote:
> > > >> > >
> > > >> > > Hi Andor,
> > > >> > >
> > > >> > > Which 2.4 release do you test in your lab? We use this feature
> > at
> > > >> production cluster with 2.4.5.
> > > >> > > At server side, we use hfile refresh instead of wal replication. I
> > > >> used to run itbll for each release with this feature enabled. How did
> > you
> > > >> find the errors, did itbll fail?
> > > >> > >
> > > >> > > Regards,
> > > >> > > Huaxiang
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > > It's what we’ve earned
> > > > Welcome, apocalypse, what’s taken you so long?
> > > > Bring us the fitting end that we’ve been counting on
> > > > - A23, Welcome, Apocalypse
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > It's what we’ve earned
> > > Welcome, apocalypse, what’s taken you so long?
> > > Bring us the fitting end that we’ve been counting on
> > > - A23, Welcome, Apocalypse
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Unrest, ignorance distilled, nihilistic imbeciles -
> It's what we’ve earned
> Welcome, apocalypse, what’s taken you so long?
> Bring us the fitting end that we’ve been counting on
> - A23, Welcome, Apocalypse
>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Andrew Purtell <ap...@apache.org>.
On Sat, Mar 26, 2022 at 10:00 PM Huaxiang Sun <hu...@apache.org>
wrote:
> It makes sense to turn on async wal replication when
> hbase.meta.replicas.use = true. Let me run couple rounds of itbll with
> hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0
> candidates) to get more confidence before proposing turn on async wal
> replication for meta.
>
+1
Thanks,
> Huaxiang
>
> On 2022/03/26 04:03:15 Andrew Purtell wrote:
> > Just to be clear when I say "it seems pointless to have meta replicas
> which
> > do not actually receive updates (by default)", what I should have said is
> > 'timely updates', because a long delay in updating meta might as well be
> a
> > missed update.
> >
> > On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org>
> wrote:
> >
> > > > "Async WAL replication for META is added as a new feature in 2.4.0.
> It
> > > is still under active development. Use with caution. Set
> > > hbase.region.replica.replication.catalog.enabled to enable async WAL
> > > Replication for META region replicas. It is off by default."
> > >
> > > Do we still need this warning?
> > >
> > > Should hbase.region.replica.replication.catalog.enabled have a default
> of
> > > 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it
> seems
> > > pointless to have meta replicas which do not actually receive updates
> (by
> > > default).
> > >
> > >
> > > On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
> > > wrote:
> > >
> > >> Hi Andor,
> > >>
> > >> I get what you are saying. The HFile refreshing is the old way for
> > >> replica regions to refresh hfiles periodically, default is 5 minutes.
> In
> > >> this itbll case, we need to have the wal replication enabled for meta
> > >> replica. Please check out,
> > >>
> > >>
> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0
> .
> > >> Basically, you need to set
> > >> "hbase.region.replica.replication.catalog.enabled" to true in the
> > >> configuration and rerun itbll. Otherwise, all meta changes at the
> primary
> > >> meta region wont be updated at the replica meta regions and it will
> result
> > >> in itbll failures.
> > >>
> > >> Hope this helps,
> > >>
> > >> Huaxiang
> > >>
> > >>
> > >> On 2022/03/25 13:46:42 Andor Molnar wrote:
> > >> > Hi Huaxiang,
> > >> >
> > >> > We use 2.4.6 for the tests.
> > >> >
> > >> > I run itbll with the following command:
> > >> >
> > >> > hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
> > >> generator 15 1000000 /tmp/hbase-itbll
> > >> >
> > >> > for the generator step and essentially jobs have failed. We can see
> the
> > >> meta request are spanning out to replicas, but writes start failing
> after
> > >> this due to the stale cache which is not getting updated.
> > >> >
> > >> > Would you please tell me more about ‘hfile refresh’ and how to
> > >> configure it?
> > >> >
> > >> > Thanks,
> > >> > Andor
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > > On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
> > >> wrote:
> > >> > >
> > >> > > Hi Andor,
> > >> > >
> > >> > > Which 2.4 release do you test in your lab? We use this feature
> at
> > >> production cluster with 2.4.5.
> > >> > > At server side, we use hfile refresh instead of wal replication. I
> > >> used to run itbll for each release with this feature enabled. How did
> you
> > >> find the errors, did itbll fail?
> > >> > >
> > >> > > Regards,
> > >> > > Huaxiang
> > >> >
> > >> >
> > >>
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > It's what we’ve earned
> > > Welcome, apocalypse, what’s taken you so long?
> > > Bring us the fitting end that we’ve been counting on
> > > - A23, Welcome, Apocalypse
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Unrest, ignorance distilled, nihilistic imbeciles -
> > It's what we’ve earned
> > Welcome, apocalypse, what’s taken you so long?
> > Bring us the fitting end that we’ve been counting on
> > - A23, Welcome, Apocalypse
> >
>
--
Best regards,
Andrew
Unrest, ignorance distilled, nihilistic imbeciles -
It's what we’ve earned
Welcome, apocalypse, what’s taken you so long?
Bring us the fitting end that we’ve been counting on
- A23, Welcome, Apocalypse
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Huaxiang Sun <hu...@apache.org>.
It makes sense to turn on async wal replication when hbase.meta.replicas.use = true. Let me run couple rounds of itbll with hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 candidates) to get more confidence before proposing turn on async wal replication for meta.
Thanks,
Huaxiang
On 2022/03/26 04:03:15 Andrew Purtell wrote:
> Just to be clear when I say "it seems pointless to have meta replicas which
> do not actually receive updates (by default)", what I should have said is
> 'timely updates', because a long delay in updating meta might as well be a
> missed update.
>
> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org> wrote:
>
> > > "Async WAL replication for META is added as a new feature in 2.4.0. It
> > is still under active development. Use with caution. Set
> > hbase.region.replica.replication.catalog.enabled to enable async WAL
> > Replication for META region replicas. It is off by default."
> >
> > Do we still need this warning?
> >
> > Should hbase.region.replica.replication.catalog.enabled have a default of
> > 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems
> > pointless to have meta replicas which do not actually receive updates (by
> > default).
> >
> >
> > On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
> > wrote:
> >
> >> Hi Andor,
> >>
> >> I get what you are saying. The HFile refreshing is the old way for
> >> replica regions to refresh hfiles periodically, default is 5 minutes. In
> >> this itbll case, we need to have the wal replication enabled for meta
> >> replica. Please check out,
> >>
> >> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0.
> >> Basically, you need to set
> >> "hbase.region.replica.replication.catalog.enabled" to true in the
> >> configuration and rerun itbll. Otherwise, all meta changes at the primary
> >> meta region wont be updated at the replica meta regions and it will result
> >> in itbll failures.
> >>
> >> Hope this helps,
> >>
> >> Huaxiang
> >>
> >>
> >> On 2022/03/25 13:46:42 Andor Molnar wrote:
> >> > Hi Huaxiang,
> >> >
> >> > We use 2.4.6 for the tests.
> >> >
> >> > I run itbll with the following command:
> >> >
> >> > hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
> >> generator 15 1000000 /tmp/hbase-itbll
> >> >
> >> > for the generator step and essentially jobs have failed. We can see the
> >> meta request are spanning out to replicas, but writes start failing after
> >> this due to the stale cache which is not getting updated.
> >> >
> >> > Would you please tell me more about ‘hfile refresh’ and how to
> >> configure it?
> >> >
> >> > Thanks,
> >> > Andor
> >> >
> >> >
> >> >
> >> >
> >> > > On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
> >> wrote:
> >> > >
> >> > > Hi Andor,
> >> > >
> >> > > Which 2.4 release do you test in your lab? We use this feature at
> >> production cluster with 2.4.5.
> >> > > At server side, we use hfile refresh instead of wal replication. I
> >> used to run itbll for each release with this feature enabled. How did you
> >> find the errors, did itbll fail?
> >> > >
> >> > > Regards,
> >> > > Huaxiang
> >> >
> >> >
> >>
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Unrest, ignorance distilled, nihilistic imbeciles -
> > It's what we’ve earned
> > Welcome, apocalypse, what’s taken you so long?
> > Bring us the fitting end that we’ve been counting on
> > - A23, Welcome, Apocalypse
> >
>
>
> --
> Best regards,
> Andrew
>
> Unrest, ignorance distilled, nihilistic imbeciles -
> It's what we’ve earned
> Welcome, apocalypse, what’s taken you so long?
> Bring us the fitting end that we’ve been counting on
> - A23, Welcome, Apocalypse
>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Andrew Purtell <ap...@apache.org>.
Just to be clear when I say "it seems pointless to have meta replicas which
do not actually receive updates (by default)", what I should have said is
'timely updates', because a long delay in updating meta might as well be a
missed update.
On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <ap...@apache.org> wrote:
> > "Async WAL replication for META is added as a new feature in 2.4.0. It
> is still under active development. Use with caution. Set
> hbase.region.replica.replication.catalog.enabled to enable async WAL
> Replication for META region replicas. It is off by default."
>
> Do we still need this warning?
>
> Should hbase.region.replica.replication.catalog.enabled have a default of
> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems
> pointless to have meta replicas which do not actually receive updates (by
> default).
>
>
> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
> wrote:
>
>> Hi Andor,
>>
>> I get what you are saying. The HFile refreshing is the old way for
>> replica regions to refresh hfiles periodically, default is 5 minutes. In
>> this itbll case, we need to have the wal replication enabled for meta
>> replica. Please check out,
>>
>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0.
>> Basically, you need to set
>> "hbase.region.replica.replication.catalog.enabled" to true in the
>> configuration and rerun itbll. Otherwise, all meta changes at the primary
>> meta region wont be updated at the replica meta regions and it will result
>> in itbll failures.
>>
>> Hope this helps,
>>
>> Huaxiang
>>
>>
>> On 2022/03/25 13:46:42 Andor Molnar wrote:
>> > Hi Huaxiang,
>> >
>> > We use 2.4.6 for the tests.
>> >
>> > I run itbll with the following command:
>> >
>> > hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
>> generator 15 1000000 /tmp/hbase-itbll
>> >
>> > for the generator step and essentially jobs have failed. We can see the
>> meta request are spanning out to replicas, but writes start failing after
>> this due to the stale cache which is not getting updated.
>> >
>> > Would you please tell me more about ‘hfile refresh’ and how to
>> configure it?
>> >
>> > Thanks,
>> > Andor
>> >
>> >
>> >
>> >
>> > > On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
>> wrote:
>> > >
>> > > Hi Andor,
>> > >
>> > > Which 2.4 release do you test in your lab? We use this feature at
>> production cluster with 2.4.5.
>> > > At server side, we use hfile refresh instead of wal replication. I
>> used to run itbll for each release with this feature enabled. How did you
>> find the errors, did itbll fail?
>> > >
>> > > Regards,
>> > > Huaxiang
>> >
>> >
>>
>
>
> --
> Best regards,
> Andrew
>
> Unrest, ignorance distilled, nihilistic imbeciles -
> It's what we’ve earned
> Welcome, apocalypse, what’s taken you so long?
> Bring us the fitting end that we’ve been counting on
> - A23, Welcome, Apocalypse
>
--
Best regards,
Andrew
Unrest, ignorance distilled, nihilistic imbeciles -
It's what we’ve earned
Welcome, apocalypse, what’s taken you so long?
Bring us the fitting end that we’ve been counting on
- A23, Welcome, Apocalypse
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Andrew Purtell <ap...@apache.org>.
> "Async WAL replication for META is added as a new feature in 2.4.0. It is
still under active development. Use with caution. Set
hbase.region.replica.replication.catalog.enabled to enable async WAL
Replication for META region replicas. It is off by default."
Do we still need this warning?
Should hbase.region.replica.replication.catalog.enabled have a default of
'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems
pointless to have meta replicas which do not actually receive updates (by
default).
On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <hu...@apache.org>
wrote:
> Hi Andor,
>
> I get what you are saying. The HFile refreshing is the old way for
> replica regions to refresh hfiles periodically, default is 5 minutes. In
> this itbll case, we need to have the wal replication enabled for meta
> replica. Please check out,
>
> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0.
> Basically, you need to set
> "hbase.region.replica.replication.catalog.enabled" to true in the
> configuration and rerun itbll. Otherwise, all meta changes at the primary
> meta region wont be updated at the replica meta regions and it will result
> in itbll failures.
>
> Hope this helps,
>
> Huaxiang
>
>
> On 2022/03/25 13:46:42 Andor Molnar wrote:
> > Hi Huaxiang,
> >
> > We use 2.4.6 for the tests.
> >
> > I run itbll with the following command:
> >
> > hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList
> generator 15 1000000 /tmp/hbase-itbll
> >
> > for the generator step and essentially jobs have failed. We can see the
> meta request are spanning out to replicas, but writes start failing after
> this due to the stale cache which is not getting updated.
> >
> > Would you please tell me more about ‘hfile refresh’ and how to configure
> it?
> >
> > Thanks,
> > Andor
> >
> >
> >
> >
> > > On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org>
> wrote:
> > >
> > > Hi Andor,
> > >
> > > Which 2.4 release do you test in your lab? We use this feature at
> production cluster with 2.4.5.
> > > At server side, we use hfile refresh instead of wal replication. I
> used to run itbll for each release with this feature enabled. How did you
> find the errors, did itbll fail?
> > >
> > > Regards,
> > > Huaxiang
> >
> >
>
--
Best regards,
Andrew
Unrest, ignorance distilled, nihilistic imbeciles -
It's what we’ve earned
Welcome, apocalypse, what’s taken you so long?
Bring us the fitting end that we’ve been counting on
- A23, Welcome, Apocalypse
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Huaxiang Sun <hu...@apache.org>.
Hi Andor,
I get what you are saying. The HFile refreshing is the old way for replica regions to refresh hfiles periodically, default is 5 minutes. In this itbll case, we need to have the wal replication enabled for meta replica. Please check out,
https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0. Basically, you need to set "hbase.region.replica.replication.catalog.enabled" to true in the configuration and rerun itbll. Otherwise, all meta changes at the primary meta region wont be updated at the replica meta regions and it will result in itbll failures.
Hope this helps,
Huaxiang
On 2022/03/25 13:46:42 Andor Molnar wrote:
> Hi Huaxiang,
>
> We use 2.4.6 for the tests.
>
> I run itbll with the following command:
>
> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator 15 1000000 /tmp/hbase-itbll
>
> for the generator step and essentially jobs have failed. We can see the meta request are spanning out to replicas, but writes start failing after this due to the stale cache which is not getting updated.
>
> Would you please tell me more about ‘hfile refresh’ and how to configure it?
>
> Thanks,
> Andor
>
>
>
>
> > On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org> wrote:
> >
> > Hi Andor,
> >
> > Which 2.4 release do you test in your lab? We use this feature at production cluster with 2.4.5.
> > At server side, we use hfile refresh instead of wal replication. I used to run itbll for each release with this feature enabled. How did you find the errors, did itbll fail?
> >
> > Regards,
> > Huaxiang
>
>
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Andor Molnar <an...@apache.org>.
Hi Huaxiang,
We use 2.4.6 for the tests.
I run itbll with the following command:
hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator 15 1000000 /tmp/hbase-itbll
for the generator step and essentially jobs have failed. We can see the meta request are spanning out to replicas, but writes start failing after this due to the stale cache which is not getting updated.
Would you please tell me more about ‘hfile refresh’ and how to configure it?
Thanks,
Andor
> On 2022. Mar 24., at 17:43, Huaxiang Sun <hu...@apache.org> wrote:
>
> Hi Andor,
>
> Which 2.4 release do you test in your lab? We use this feature at production cluster with 2.4.5.
> At server side, we use hfile refresh instead of wal replication. I used to run itbll for each release with this feature enabled. How did you find the errors, did itbll fail?
>
> Regards,
> Huaxiang
Re: Meta replicas in LoadBalance mode broken in 2.4?
Posted by Huaxiang Sun <hu...@apache.org>.
Hi Andor,
Which 2.4 release do you test in your lab? We use this feature at production cluster with 2.4.5.
At server side, we use hfile refresh instead of wal replication. I used to run itbll for each release with this feature enabled. How did you find the errors, did itbll fail?
Regards,
Huaxiang