You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Sawant, Chandramohan " <ch...@citi.com.INVALID> on 2017/12/27 21:00:01 UTC
Region count for a table not matching between prod and cob where
one way replication is enabled from prod to cob
Hi All,
Before enabling replication, snapshot of PROD table taken and restored in COB and then replication enabled, region count was matching at that time.
However after few days, COB showing extra regions than in prod where replication enabled one way from PROD to COB.
What is the reason of having mismatch in the number of regions between prod and cob ?
COB -
Namespace Table Name Online Regions Offline Regions Failed Regions Split Regions Other Regions
default TABLE1 4 0 0 3 0
default TABLE2 10 0 0 8 0
PROD -
Namespace Table Name Online Regions Offline Regions Failed Regions Split Regions Other Regions
default TABLE1 1 0 0 0 0
default TABLE2 4 0 0 2 0
hbase:meta table in COB shows the few regions of above tables are offline with offline attribute true, however it shows different status on HBase UI, please see above.
Regards,
CM
+1 201 763 1656
RE: Region count for a table not matching between prod and cob
where one way replication is enabled from prod to cob
Posted by "Sawant, Chandramohan " <ch...@citi.com.INVALID>.
Thanks Sudhir and Ted,
Please find below the response -
hbase.hregion.max.filesize carry same value in both clusters, it is set to 10 Gigs.
That is true, there are 3 splits in COB and I can see that in hbase:meta table.
As per my understanding, COB cluster should have lesser online regions than PROD because replication might be taking only latest WALs over the time period while replicating to COB.
In COB, if I run the row counter map reduce job on snapshot taken on table, it gathers records from offline regions too and leads to incorrect extra row count.
Regards,
CM
+1 201 763 1656
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Wednesday, December 27, 2017 6:58 PM
To: user@hbase.apache.org
Subject: Re: Region count for a table not matching between prod and cob where one way replication is enabled from prod to cob
CM:
Does hbase.hregion.max.filesize carry same value in both clusters ?
It seems TABLE1 went thru 3 splits in COB cluster.
You can check region server log(s) to confirm the split.
Cheers
On Wed, Dec 27, 2017 at 2:32 PM, sudhir patil <sp...@gmail.com>
wrote:
> Regions in both clusters need not have to match, each server independently
> manages its regions depending on split, compactions etc number of regions
> in both servers won't be same. If you want to verify both servers are
> replicated correctly you have to run below command
>
> sudo -u hbase hbase
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication peer1
> table1
>
>
> On Thu, Dec 28, 2017 at 5:00 AM, Sawant, Chandramohan <
> chandramohan.sawant@citi.com.invalid> wrote:
>
> > Hi All,
> >
> > Before enabling replication, snapshot of PROD table taken and restored in
> > COB and then replication enabled, region count was matching at that time.
> > However after few days, COB showing extra regions than in prod where
> > replication enabled one way from PROD to COB.
> > What is the reason of having mismatch in the number of regions between
> > prod and cob ?
> >
> > COB -
> > Namespace Table Name Online Regions Offline Regions Failed
> > Regions Split Regions Other Regions
> > default TABLE1 4
> > 0 0 3
> > 0
> > default TABLE2 10
> > 0 0 8
> > 0
> >
> > PROD -
> > Namespace Table Name Online Regions Offline Regions Failed
> > Regions Split Regions Other Regions
> > default TABLE1 1
> > 0 0 0
> > 0
> > default TABLE2 4
> > 0 0 2
> > 0
> >
> > hbase:meta table in COB shows the few regions of above tables are offline
> > with offline attribute true, however it shows different status on HBase
> UI,
> > please see above.
> >
> > Regards,
> > CM
> > +1 201 763 1656
> >
> >
>
Re: Region count for a table not matching between prod and cob where
one way replication is enabled from prod to cob
Posted by Ted Yu <yu...@gmail.com>.
CM:
Does hbase.hregion.max.filesize carry same value in both clusters ?
It seems TABLE1 went thru 3 splits in COB cluster.
You can check region server log(s) to confirm the split.
Cheers
On Wed, Dec 27, 2017 at 2:32 PM, sudhir patil <sp...@gmail.com>
wrote:
> Regions in both clusters need not have to match, each server independently
> manages its regions depending on split, compactions etc number of regions
> in both servers won't be same. If you want to verify both servers are
> replicated correctly you have to run below command
>
> sudo -u hbase hbase
> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication peer1
> table1
>
>
> On Thu, Dec 28, 2017 at 5:00 AM, Sawant, Chandramohan <
> chandramohan.sawant@citi.com.invalid> wrote:
>
> > Hi All,
> >
> > Before enabling replication, snapshot of PROD table taken and restored in
> > COB and then replication enabled, region count was matching at that time.
> > However after few days, COB showing extra regions than in prod where
> > replication enabled one way from PROD to COB.
> > What is the reason of having mismatch in the number of regions between
> > prod and cob ?
> >
> > COB -
> > Namespace Table Name Online Regions Offline Regions Failed
> > Regions Split Regions Other Regions
> > default TABLE1 4
> > 0 0 3
> > 0
> > default TABLE2 10
> > 0 0 8
> > 0
> >
> > PROD -
> > Namespace Table Name Online Regions Offline Regions Failed
> > Regions Split Regions Other Regions
> > default TABLE1 1
> > 0 0 0
> > 0
> > default TABLE2 4
> > 0 0 2
> > 0
> >
> > hbase:meta table in COB shows the few regions of above tables are offline
> > with offline attribute true, however it shows different status on HBase
> UI,
> > please see above.
> >
> > Regards,
> > CM
> > +1 201 763 1656
> >
> >
>
Re: Region count for a table not matching between prod and cob where
one way replication is enabled from prod to cob
Posted by sudhir patil <sp...@gmail.com>.
Regions in both clusters need not have to match, each server independently
manages its regions depending on split, compactions etc number of regions
in both servers won't be same. If you want to verify both servers are
replicated correctly you have to run below command
sudo -u hbase hbase
org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication peer1
table1
On Thu, Dec 28, 2017 at 5:00 AM, Sawant, Chandramohan <
chandramohan.sawant@citi.com.invalid> wrote:
> Hi All,
>
> Before enabling replication, snapshot of PROD table taken and restored in
> COB and then replication enabled, region count was matching at that time.
> However after few days, COB showing extra regions than in prod where
> replication enabled one way from PROD to COB.
> What is the reason of having mismatch in the number of regions between
> prod and cob ?
>
> COB -
> Namespace Table Name Online Regions Offline Regions Failed
> Regions Split Regions Other Regions
> default TABLE1 4
> 0 0 3
> 0
> default TABLE2 10
> 0 0 8
> 0
>
> PROD -
> Namespace Table Name Online Regions Offline Regions Failed
> Regions Split Regions Other Regions
> default TABLE1 1
> 0 0 0
> 0
> default TABLE2 4
> 0 0 2
> 0
>
> hbase:meta table in COB shows the few regions of above tables are offline
> with offline attribute true, however it shows different status on HBase UI,
> please see above.
>
> Regards,
> CM
> +1 201 763 1656
>
>