You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jerry Lam <ch...@gmail.com> on 2012/04/20 14:38:40 UTC
HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Hi HBase community:
We have been testing cyclic replication for 1 week. The basic functionality seems to work as described in the document however when we started to increase the write workload, the replication starts to miss data (i.e. some data are not replicated to the other cluster). We have narrowed down to a scenario that we can reproduce the problem quite consistently and here it is:
-----------------------------
Setup:
- We have setup 2 clusters (cluster A and cluster B)with identical size in terms of number of nodes and configuration, 3 regionservers sit on top of 3 datanodes.
- Cyclic replication is enabled.
- We use YCSB to generate load to hbase the workload is very similar to workloada:
recordcount=200000
operationcount=200000
workload=com.yahoo.ycsb.workloads.CoreWorkload
fieldcount=1
fieldlength=25000
readallfields=true
writeallfields=true
readproportion=0
updateproportion=1
scanproportion=0
insertproportion=0
requestdistribution=uniform
- Records are inserted into Cluster A. After the benchmark is done and wait until all data are replicated to Cluster B, we used verifyrep mapreduce job for validation.
- Data are deleted from both table (truncate 'tablename') before a new experiment is started.
Scenario:
when we increase the number of threads until it max out the throughput of the cluster, we saw some data are missing in Cluster B (total count != 200000) although cluster A clearly has them all. This happens even though we disabled region splitting in both clusters (it happens more often when region splits occur). To further having more control of what is happening, we then decided to disable the load balancer so the region (which is responsible for the replicating data) will not relocate to other regionserver during the benchmark. The situation improves a lot. We don't see any missing data in 5 continuous runs. Finally, we decided to move the region around from a regionserver to another regionserver during the benchmark to see if the problem will reappear and it did.
We believe that the issue could be related to region splitting and load balancing during intensive write, the hbase replication strategy hasn't yet cover those corner cases.
Can someone take a look of it and suggest some ways to workaround this?
Thanks~
Jerry
Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Posted by Jerry Lam <ch...@gmail.com>.
Hi Himanshu:
My team is particularly interested in the cyclic replication so I have enable the master-master replication (so each cluster has the other cluster as its replication peer), although the replication was one direction (from cluster A to cluster B) in the test. I didn't stop_replication on the other cluster if that is what you mean by disabling the replication.
Thanks!
Jerry
On 2012-05-01, at 10:08 PM, Himanshu Vashishtha wrote:
> Yeah, I should have mentioned that: its master-master, and on cdh4b1.
> But, replication on that specific slave table is disabled (so,
> effectively its master-slave for this test).
>
> Is this same as yours (replication config wise), or shall I enable
> replication on the destination table too?
>
> Thanks,
> Himanshu
>
> On Tue, May 1, 2012 at 8:01 PM, Jerry Lam <ch...@gmail.com> wrote:
>> Hi Himanshu:
>>
>> Thanks for following up! I did looked up the log and there were some exceptions. I'm not sure if those exceptions contribute to the problem I've seen a week ago.
>> I did aware of the latency between the time that the master said "Nothing to replicate" and the actual time it takes to actually replicate on the slave. I remember I wait 12 hours for the replication to finish (i.e. start the test before leaving office and check the result the next day) and data still not fully replicated.
>>
>> By the way, is your test running with master-slave replication or master-master replication?
>>
>> I will resume this again. I was busy on something else for the past week or so.
>>
>> Best Regards,
>>
>> Jerry
>>
>> On 2012-05-01, at 6:41 PM, Himanshu Vashishtha wrote:
>>
>>> Hello Jerry,
>>>
>>> Did you try this again.
>>>
>>> Whenever you try next, can you please share the logs somehow.
>>>
>>> I tried replicating your scenario today, but no luck. I used the same
>>> workload you have copied here; master cluster has 5 nodes and slave
>>> has just 2 nodes; and made tiny regions of 8MB (memstore flushing at
>>> 8mb too), so that I have around 1200+ regions even for 200k rows; ran
>>> the workload with 16, 24 and 32 client threads, but the verifyrep
>>> mapreduce job says its good.
>>> Yes, I ran the verifyrep command after seeing "there is nothing to
>>> replicate" message on all the regionservers; sometimes it was a bit
>>> slow.
>>>
>>>
>>> Thanks,
>>> Himanshu
>>>
>>> On Mon, Apr 23, 2012 at 11:57 AM, Jean-Daniel Cryans
>>> <jd...@apache.org> wrote:
>>>>> I will try your suggestion today with a master-slave replication enabled from Cluster A -> Cluster B.
>>>>
>>>> Please do.
>>>>
>>>>> Last Friday, I tried to limit the variability/the moving part of the replication components. I reduced the size of Cluster B to have only 1 regionserver and having Cluster A to replicate data from one region only without region splitting (therefore I have 1-to-1 region replication setup). During the benchmark, I moved the region between different regionservers in Cluster A (note there are still 3 regionservers in Cluster A). I ran this test for 5 times and no data were lost. Does it mean something? My feeling is there are some glitches/corner cases that have not been covered in the cyclic replication (or hbase replication in general). Note that, this happens only when the load is high.
>>>>
>>>> And have you looked at the logs? Any obvious exceptions coming up?
>>>> Replication uses the normal HBase client to insert the data on the
>>>> other cluster and this is what handles regions moving around.
>>>>
>>>>>
>>>>> By the way, why do we need to have a zookeeper not handled by hbase for the replication to work (it is described in the hbase documentation)?
>>>>
>>>> It says you *should* do it, not you *need* to do it :)
>>>>
>>>> But basically replication is zk-heavy and getting a better
>>>> understanding of it starts with handling it yourself.
>>>>
>>>> J-D
>>
Re: HBase Cyclic Replication Issue: some data are missing in the
replication for intensive write
Posted by Himanshu Vashishtha <hv...@cs.ualberta.ca>.
Yeah, I should have mentioned that: its master-master, and on cdh4b1.
But, replication on that specific slave table is disabled (so,
effectively its master-slave for this test).
Is this same as yours (replication config wise), or shall I enable
replication on the destination table too?
Thanks,
Himanshu
On Tue, May 1, 2012 at 8:01 PM, Jerry Lam <ch...@gmail.com> wrote:
> Hi Himanshu:
>
> Thanks for following up! I did looked up the log and there were some exceptions. I'm not sure if those exceptions contribute to the problem I've seen a week ago.
> I did aware of the latency between the time that the master said "Nothing to replicate" and the actual time it takes to actually replicate on the slave. I remember I wait 12 hours for the replication to finish (i.e. start the test before leaving office and check the result the next day) and data still not fully replicated.
>
> By the way, is your test running with master-slave replication or master-master replication?
>
> I will resume this again. I was busy on something else for the past week or so.
>
> Best Regards,
>
> Jerry
>
> On 2012-05-01, at 6:41 PM, Himanshu Vashishtha wrote:
>
>> Hello Jerry,
>>
>> Did you try this again.
>>
>> Whenever you try next, can you please share the logs somehow.
>>
>> I tried replicating your scenario today, but no luck. I used the same
>> workload you have copied here; master cluster has 5 nodes and slave
>> has just 2 nodes; and made tiny regions of 8MB (memstore flushing at
>> 8mb too), so that I have around 1200+ regions even for 200k rows; ran
>> the workload with 16, 24 and 32 client threads, but the verifyrep
>> mapreduce job says its good.
>> Yes, I ran the verifyrep command after seeing "there is nothing to
>> replicate" message on all the regionservers; sometimes it was a bit
>> slow.
>>
>>
>> Thanks,
>> Himanshu
>>
>> On Mon, Apr 23, 2012 at 11:57 AM, Jean-Daniel Cryans
>> <jd...@apache.org> wrote:
>>>> I will try your suggestion today with a master-slave replication enabled from Cluster A -> Cluster B.
>>>
>>> Please do.
>>>
>>>> Last Friday, I tried to limit the variability/the moving part of the replication components. I reduced the size of Cluster B to have only 1 regionserver and having Cluster A to replicate data from one region only without region splitting (therefore I have 1-to-1 region replication setup). During the benchmark, I moved the region between different regionservers in Cluster A (note there are still 3 regionservers in Cluster A). I ran this test for 5 times and no data were lost. Does it mean something? My feeling is there are some glitches/corner cases that have not been covered in the cyclic replication (or hbase replication in general). Note that, this happens only when the load is high.
>>>
>>> And have you looked at the logs? Any obvious exceptions coming up?
>>> Replication uses the normal HBase client to insert the data on the
>>> other cluster and this is what handles regions moving around.
>>>
>>>>
>>>> By the way, why do we need to have a zookeeper not handled by hbase for the replication to work (it is described in the hbase documentation)?
>>>
>>> It says you *should* do it, not you *need* to do it :)
>>>
>>> But basically replication is zk-heavy and getting a better
>>> understanding of it starts with handling it yourself.
>>>
>>> J-D
>
Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Posted by Jerry Lam <ch...@gmail.com>.
Hi Himanshu:
Thanks for following up! I did looked up the log and there were some exceptions. I'm not sure if those exceptions contribute to the problem I've seen a week ago.
I did aware of the latency between the time that the master said "Nothing to replicate" and the actual time it takes to actually replicate on the slave. I remember I wait 12 hours for the replication to finish (i.e. start the test before leaving office and check the result the next day) and data still not fully replicated.
By the way, is your test running with master-slave replication or master-master replication?
I will resume this again. I was busy on something else for the past week or so.
Best Regards,
Jerry
On 2012-05-01, at 6:41 PM, Himanshu Vashishtha wrote:
> Hello Jerry,
>
> Did you try this again.
>
> Whenever you try next, can you please share the logs somehow.
>
> I tried replicating your scenario today, but no luck. I used the same
> workload you have copied here; master cluster has 5 nodes and slave
> has just 2 nodes; and made tiny regions of 8MB (memstore flushing at
> 8mb too), so that I have around 1200+ regions even for 200k rows; ran
> the workload with 16, 24 and 32 client threads, but the verifyrep
> mapreduce job says its good.
> Yes, I ran the verifyrep command after seeing "there is nothing to
> replicate" message on all the regionservers; sometimes it was a bit
> slow.
>
>
> Thanks,
> Himanshu
>
> On Mon, Apr 23, 2012 at 11:57 AM, Jean-Daniel Cryans
> <jd...@apache.org> wrote:
>>> I will try your suggestion today with a master-slave replication enabled from Cluster A -> Cluster B.
>>
>> Please do.
>>
>>> Last Friday, I tried to limit the variability/the moving part of the replication components. I reduced the size of Cluster B to have only 1 regionserver and having Cluster A to replicate data from one region only without region splitting (therefore I have 1-to-1 region replication setup). During the benchmark, I moved the region between different regionservers in Cluster A (note there are still 3 regionservers in Cluster A). I ran this test for 5 times and no data were lost. Does it mean something? My feeling is there are some glitches/corner cases that have not been covered in the cyclic replication (or hbase replication in general). Note that, this happens only when the load is high.
>>
>> And have you looked at the logs? Any obvious exceptions coming up?
>> Replication uses the normal HBase client to insert the data on the
>> other cluster and this is what handles regions moving around.
>>
>>>
>>> By the way, why do we need to have a zookeeper not handled by hbase for the replication to work (it is described in the hbase documentation)?
>>
>> It says you *should* do it, not you *need* to do it :)
>>
>> But basically replication is zk-heavy and getting a better
>> understanding of it starts with handling it yourself.
>>
>> J-D
Re: HBase Cyclic Replication Issue: some data are missing in the
replication for intensive write
Posted by Himanshu Vashishtha <hv...@cs.ualberta.ca>.
Hello Jerry,
Did you try this again.
Whenever you try next, can you please share the logs somehow.
I tried replicating your scenario today, but no luck. I used the same
workload you have copied here; master cluster has 5 nodes and slave
has just 2 nodes; and made tiny regions of 8MB (memstore flushing at
8mb too), so that I have around 1200+ regions even for 200k rows; ran
the workload with 16, 24 and 32 client threads, but the verifyrep
mapreduce job says its good.
Yes, I ran the verifyrep command after seeing "there is nothing to
replicate" message on all the regionservers; sometimes it was a bit
slow.
Thanks,
Himanshu
On Mon, Apr 23, 2012 at 11:57 AM, Jean-Daniel Cryans
<jd...@apache.org> wrote:
>> I will try your suggestion today with a master-slave replication enabled from Cluster A -> Cluster B.
>
> Please do.
>
>> Last Friday, I tried to limit the variability/the moving part of the replication components. I reduced the size of Cluster B to have only 1 regionserver and having Cluster A to replicate data from one region only without region splitting (therefore I have 1-to-1 region replication setup). During the benchmark, I moved the region between different regionservers in Cluster A (note there are still 3 regionservers in Cluster A). I ran this test for 5 times and no data were lost. Does it mean something? My feeling is there are some glitches/corner cases that have not been covered in the cyclic replication (or hbase replication in general). Note that, this happens only when the load is high.
>
> And have you looked at the logs? Any obvious exceptions coming up?
> Replication uses the normal HBase client to insert the data on the
> other cluster and this is what handles regions moving around.
>
>>
>> By the way, why do we need to have a zookeeper not handled by hbase for the replication to work (it is described in the hbase documentation)?
>
> It says you *should* do it, not you *need* to do it :)
>
> But basically replication is zk-heavy and getting a better
> understanding of it starts with handling it yourself.
>
> J-D
Re: HBase Cyclic Replication Issue: some data are missing in the
replication for intensive write
Posted by Jean-Daniel Cryans <jd...@apache.org>.
> I will try your suggestion today with a master-slave replication enabled from Cluster A -> Cluster B.
Please do.
> Last Friday, I tried to limit the variability/the moving part of the replication components. I reduced the size of Cluster B to have only 1 regionserver and having Cluster A to replicate data from one region only without region splitting (therefore I have 1-to-1 region replication setup). During the benchmark, I moved the region between different regionservers in Cluster A (note there are still 3 regionservers in Cluster A). I ran this test for 5 times and no data were lost. Does it mean something? My feeling is there are some glitches/corner cases that have not been covered in the cyclic replication (or hbase replication in general). Note that, this happens only when the load is high.
And have you looked at the logs? Any obvious exceptions coming up?
Replication uses the normal HBase client to insert the data on the
other cluster and this is what handles regions moving around.
>
> By the way, why do we need to have a zookeeper not handled by hbase for the replication to work (it is described in the hbase documentation)?
It says you *should* do it, not you *need* to do it :)
But basically replication is zk-heavy and getting a better
understanding of it starts with handling it yourself.
J-D
Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Posted by Jerry Lam <ch...@gmail.com>.
Hi Lars:
I will try your suggestion today with a master-slave replication enabled from Cluster A -> Cluster B.
Last Friday, I tried to limit the variability/the moving part of the replication components. I reduced the size of Cluster B to have only 1 regionserver and having Cluster A to replicate data from one region only without region splitting (therefore I have 1-to-1 region replication setup). During the benchmark, I moved the region between different regionservers in Cluster A (note there are still 3 regionservers in Cluster A). I ran this test for 5 times and no data were lost. Does it mean something? My feeling is there are some glitches/corner cases that have not been covered in the cyclic replication (or hbase replication in general). Note that, this happens only when the load is high.
By the way, why do we need to have a zookeeper not handled by hbase for the replication to work (it is described in the hbase documentation)?
Best Regards,
Jerry
On 2012-04-20, at 7:08 PM, lars hofhansl wrote:
> I see.
> Does this only happen when cyclic replication is enabled in this way (i.e. master <-> master replication).
> The replication back does take some overhead as the replicator needs to filter edits from being replication back to the originator, but I would not have thought that would cause any issues.
>
> Could you run the same test once with replication only enabled from ClusterA -> ClusterB?
>
> Thanks.
>
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Jerry Lam <ch...@gmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Cc:
> Sent: Friday, April 20, 2012 3:43 PM
> Subject: Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
>
> Hi Himanshu:
>
> I'm using hbase 0.92.1 and hadoop 1.0.1 migrating from hbase 0.90.4 and Hadoop 0.20 with append feature.
>
> It is a one side replication (cluster A to cluster B) with cyclic replication enabled (i.e. add_peer of the other cluster configured).
>
> Best Regards,
>
> Jerry
>
> Sent from my iPad
>
> On 2012-04-20, at 10:23, Himanshu Vashishtha <hv...@cs.ualberta.ca> wrote:
>
>> Hello Jerry,
>>
>> Which HBase version?
>>
>> You are not "using" cyclic replication? Its simple one side replication, right?
>>
>> Thanks,
>> Himanshu
>>
>> On Fri, Apr 20, 2012 at 5:38 AM, Jerry Lam <ch...@gmail.com> wrote:
>>> Hi HBase community:
>>>
>>> We have been testing cyclic replication for 1 week. The basic functionality seems to work as described in the document however when we started to increase the write workload, the replication starts to miss data (i.e. some data are not replicated to the other cluster). We have narrowed down to a scenario that we can reproduce the problem quite consistently and here it is:
>>>
>>> -----------------------------
>>> Setup:
>>> - We have setup 2 clusters (cluster A and cluster B)with identical size in terms of number of nodes and configuration, 3 regionservers sit on top of 3 datanodes.
>>> - Cyclic replication is enabled.
>>>
>>> - We use YCSB to generate load to hbase the workload is very similar to workloada:
>>>
>>> recordcount=200000
>>> operationcount=200000
>>> workload=com.yahoo.ycsb.workloads.CoreWorkload
>>> fieldcount=1
>>> fieldlength=25000
>>>
>>> readallfields=true
>>> writeallfields=true
>>>
>>> readproportion=0
>>> updateproportion=1
>>> scanproportion=0
>>> insertproportion=0
>>>
>>> requestdistribution=uniform
>>>
>>> - Records are inserted into Cluster A. After the benchmark is done and wait until all data are replicated to Cluster B, we used verifyrep mapreduce job for validation.
>>> - Data are deleted from both table (truncate 'tablename') before a new experiment is started.
>>>
>>> Scenario:
>>> when we increase the number of threads until it max out the throughput of the cluster, we saw some data are missing in Cluster B (total count != 200000) although cluster A clearly has them all. This happens even though we disabled region splitting in both clusters (it happens more often when region splits occur). To further having more control of what is happening, we then decided to disable the load balancer so the region (which is responsible for the replicating data) will not relocate to other regionserver during the benchmark. The situation improves a lot. We don't see any missing data in 5 continuous runs. Finally, we decided to move the region around from a regionserver to another regionserver during the benchmark to see if the problem will reappear and it did.
>>>
>>> We believe that the issue could be related to region splitting and load balancing during intensive write, the hbase replication strategy hasn't yet cover those corner cases.
>>>
>>> Can someone take a look of it and suggest some ways to workaround this?
>>>
>>> Thanks~
>>>
>>> Jerry
>
Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Posted by lars hofhansl <lh...@yahoo.com>.
I see.
Does this only happen when cyclic replication is enabled in this way (i.e. master <-> master replication).
The replication back does take some overhead as the replicator needs to filter edits from being replication back to the originator, but I would not have thought that would cause any issues.
Could you run the same test once with replication only enabled from ClusterA -> ClusterB?
Thanks.
-- Lars
----- Original Message -----
From: Jerry Lam <ch...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc:
Sent: Friday, April 20, 2012 3:43 PM
Subject: Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Hi Himanshu:
I'm using hbase 0.92.1 and hadoop 1.0.1 migrating from hbase 0.90.4 and Hadoop 0.20 with append feature.
It is a one side replication (cluster A to cluster B) with cyclic replication enabled (i.e. add_peer of the other cluster configured).
Best Regards,
Jerry
Sent from my iPad
On 2012-04-20, at 10:23, Himanshu Vashishtha <hv...@cs.ualberta.ca> wrote:
> Hello Jerry,
>
> Which HBase version?
>
> You are not "using" cyclic replication? Its simple one side replication, right?
>
> Thanks,
> Himanshu
>
> On Fri, Apr 20, 2012 at 5:38 AM, Jerry Lam <ch...@gmail.com> wrote:
>> Hi HBase community:
>>
>> We have been testing cyclic replication for 1 week. The basic functionality seems to work as described in the document however when we started to increase the write workload, the replication starts to miss data (i.e. some data are not replicated to the other cluster). We have narrowed down to a scenario that we can reproduce the problem quite consistently and here it is:
>>
>> -----------------------------
>> Setup:
>> - We have setup 2 clusters (cluster A and cluster B)with identical size in terms of number of nodes and configuration, 3 regionservers sit on top of 3 datanodes.
>> - Cyclic replication is enabled.
>>
>> - We use YCSB to generate load to hbase the workload is very similar to workloada:
>>
>> recordcount=200000
>> operationcount=200000
>> workload=com.yahoo.ycsb.workloads.CoreWorkload
>> fieldcount=1
>> fieldlength=25000
>>
>> readallfields=true
>> writeallfields=true
>>
>> readproportion=0
>> updateproportion=1
>> scanproportion=0
>> insertproportion=0
>>
>> requestdistribution=uniform
>>
>> - Records are inserted into Cluster A. After the benchmark is done and wait until all data are replicated to Cluster B, we used verifyrep mapreduce job for validation.
>> - Data are deleted from both table (truncate 'tablename') before a new experiment is started.
>>
>> Scenario:
>> when we increase the number of threads until it max out the throughput of the cluster, we saw some data are missing in Cluster B (total count != 200000) although cluster A clearly has them all. This happens even though we disabled region splitting in both clusters (it happens more often when region splits occur). To further having more control of what is happening, we then decided to disable the load balancer so the region (which is responsible for the replicating data) will not relocate to other regionserver during the benchmark. The situation improves a lot. We don't see any missing data in 5 continuous runs. Finally, we decided to move the region around from a regionserver to another regionserver during the benchmark to see if the problem will reappear and it did.
>>
>> We believe that the issue could be related to region splitting and load balancing during intensive write, the hbase replication strategy hasn't yet cover those corner cases.
>>
>> Can someone take a look of it and suggest some ways to workaround this?
>>
>> Thanks~
>>
>> Jerry
Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Posted by Jerry Lam <ch...@gmail.com>.
Hi Himanshu:
I'm using hbase 0.92.1 and hadoop 1.0.1 migrating from hbase 0.90.4 and Hadoop 0.20 with append feature.
It is a one side replication (cluster A to cluster B) with cyclic replication enabled (i.e. add_peer of the other cluster configured).
Best Regards,
Jerry
Sent from my iPad
On 2012-04-20, at 10:23, Himanshu Vashishtha <hv...@cs.ualberta.ca> wrote:
> Hello Jerry,
>
> Which HBase version?
>
> You are not "using" cyclic replication? Its simple one side replication, right?
>
> Thanks,
> Himanshu
>
> On Fri, Apr 20, 2012 at 5:38 AM, Jerry Lam <ch...@gmail.com> wrote:
>> Hi HBase community:
>>
>> We have been testing cyclic replication for 1 week. The basic functionality seems to work as described in the document however when we started to increase the write workload, the replication starts to miss data (i.e. some data are not replicated to the other cluster). We have narrowed down to a scenario that we can reproduce the problem quite consistently and here it is:
>>
>> -----------------------------
>> Setup:
>> - We have setup 2 clusters (cluster A and cluster B)with identical size in terms of number of nodes and configuration, 3 regionservers sit on top of 3 datanodes.
>> - Cyclic replication is enabled.
>>
>> - We use YCSB to generate load to hbase the workload is very similar to workloada:
>>
>> recordcount=200000
>> operationcount=200000
>> workload=com.yahoo.ycsb.workloads.CoreWorkload
>> fieldcount=1
>> fieldlength=25000
>>
>> readallfields=true
>> writeallfields=true
>>
>> readproportion=0
>> updateproportion=1
>> scanproportion=0
>> insertproportion=0
>>
>> requestdistribution=uniform
>>
>> - Records are inserted into Cluster A. After the benchmark is done and wait until all data are replicated to Cluster B, we used verifyrep mapreduce job for validation.
>> - Data are deleted from both table (truncate 'tablename') before a new experiment is started.
>>
>> Scenario:
>> when we increase the number of threads until it max out the throughput of the cluster, we saw some data are missing in Cluster B (total count != 200000) although cluster A clearly has them all. This happens even though we disabled region splitting in both clusters (it happens more often when region splits occur). To further having more control of what is happening, we then decided to disable the load balancer so the region (which is responsible for the replicating data) will not relocate to other regionserver during the benchmark. The situation improves a lot. We don't see any missing data in 5 continuous runs. Finally, we decided to move the region around from a regionserver to another regionserver during the benchmark to see if the problem will reappear and it did.
>>
>> We believe that the issue could be related to region splitting and load balancing during intensive write, the hbase replication strategy hasn't yet cover those corner cases.
>>
>> Can someone take a look of it and suggest some ways to workaround this?
>>
>> Thanks~
>>
>> Jerry
Re: HBase Cyclic Replication Issue: some data are missing in the
replication for intensive write
Posted by Himanshu Vashishtha <hv...@cs.ualberta.ca>.
Hello Jerry,
Which HBase version?
You are not "using" cyclic replication? Its simple one side replication, right?
Thanks,
Himanshu
On Fri, Apr 20, 2012 at 5:38 AM, Jerry Lam <ch...@gmail.com> wrote:
> Hi HBase community:
>
> We have been testing cyclic replication for 1 week. The basic functionality seems to work as described in the document however when we started to increase the write workload, the replication starts to miss data (i.e. some data are not replicated to the other cluster). We have narrowed down to a scenario that we can reproduce the problem quite consistently and here it is:
>
> -----------------------------
> Setup:
> - We have setup 2 clusters (cluster A and cluster B)with identical size in terms of number of nodes and configuration, 3 regionservers sit on top of 3 datanodes.
> - Cyclic replication is enabled.
>
> - We use YCSB to generate load to hbase the workload is very similar to workloada:
>
> recordcount=200000
> operationcount=200000
> workload=com.yahoo.ycsb.workloads.CoreWorkload
> fieldcount=1
> fieldlength=25000
>
> readallfields=true
> writeallfields=true
>
> readproportion=0
> updateproportion=1
> scanproportion=0
> insertproportion=0
>
> requestdistribution=uniform
>
> - Records are inserted into Cluster A. After the benchmark is done and wait until all data are replicated to Cluster B, we used verifyrep mapreduce job for validation.
> - Data are deleted from both table (truncate 'tablename') before a new experiment is started.
>
> Scenario:
> when we increase the number of threads until it max out the throughput of the cluster, we saw some data are missing in Cluster B (total count != 200000) although cluster A clearly has them all. This happens even though we disabled region splitting in both clusters (it happens more often when region splits occur). To further having more control of what is happening, we then decided to disable the load balancer so the region (which is responsible for the replicating data) will not relocate to other regionserver during the benchmark. The situation improves a lot. We don't see any missing data in 5 continuous runs. Finally, we decided to move the region around from a regionserver to another regionserver during the benchmark to see if the problem will reappear and it did.
>
> We believe that the issue could be related to region splitting and load balancing during intensive write, the hbase replication strategy hasn't yet cover those corner cases.
>
> Can someone take a look of it and suggest some ways to workaround this?
>
> Thanks~
>
> Jerry
Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Posted by Jerry Lam <ch...@gmail.com>.
Hi Lars:
I'm using hbase 0.92.1 and Hadoop 1.0.1.
Yes, you are right. I'm replicating from cluster A to cluster B only with cyclic replication configured. Eventually I will test replicating cluster A to cluster B and vice versa with high intensive write workload but if this replication doesn't work for one way, we need to think about other solutions.
No data loss in cluster A for sure.
Best Regards,
Jerry
Sent from my iPad
On 2012-04-20, at 15:34, lars hofhansl <lh...@yahoo.com> wrote:
> Hi Jerry,
>
> which version of HBase are you using?
>
> You are not using cyclic backup, that needs >2 clusters. I assume you're just replicating from one cluster to another, right?
>
> There is never data loss in Cluster A?
>
> -- Lars
>
>
> ----- Original Message -----
> From: Jerry Lam <ch...@gmail.com>
> To: user@hbase.apache.org
> Cc:
> Sent: Friday, April 20, 2012 5:38 AM
> Subject: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
>
> Hi HBase community:
>
> We have been testing cyclic replication for 1 week. The basic functionality seems to work as described in the document however when we started to increase the write workload, the replication starts to miss data (i.e. some data are not replicated to the other cluster). We have narrowed down to a scenario that we can reproduce the problem quite consistently and here it is:
>
> -----------------------------
> Setup:
> - We have setup 2 clusters (cluster A and cluster B)with identical size in terms of number of nodes and configuration, 3 regionservers sit on top of 3 datanodes.
> - Cyclic replication is enabled.
>
> - We use YCSB to generate load to hbase the workload is very similar to workloada:
>
> recordcount=200000
> operationcount=200000
> workload=com.yahoo.ycsb.workloads.CoreWorkload
> fieldcount=1
> fieldlength=25000
>
> readallfields=true
> writeallfields=true
>
> readproportion=0
> updateproportion=1
> scanproportion=0
> insertproportion=0
>
> requestdistribution=uniform
>
> - Records are inserted into Cluster A. After the benchmark is done and wait until all data are replicated to Cluster B, we used verifyrep mapreduce job for validation.
> - Data are deleted from both table (truncate 'tablename') before a new experiment is started.
>
> Scenario:
> when we increase the number of threads until it max out the throughput of the cluster, we saw some data are missing in Cluster B (total count != 200000) although cluster A clearly has them all. This happens even though we disabled region splitting in both clusters (it happens more often when region splits occur). To further having more control of what is happening, we then decided to disable the load balancer so the region (which is responsible for the replicating data) will not relocate to other regionserver during the benchmark. The situation improves a lot. We don't see any missing data in 5 continuous runs. Finally, we decided to move the region around from a regionserver to another regionserver during the benchmark to see if the problem will reappear and it did.
>
> We believe that the issue could be related to region splitting and load balancing during intensive write, the hbase replication strategy hasn't yet cover those corner cases.
>
> Can someone take a look of it and suggest some ways to workaround this?
>
> Thanks~
>
> Jerry
Re: HBase Cyclic Replication Issue: some data are missing in the
replication for intensive write
Posted by Ted Yu <yu...@gmail.com>.
Jerry:
Currently TestReplicationPeer and TestReplication don't involve load
balancing scenario.
If you can write a test where load balancer re-assigns some regions, that
would help us pinpoint the problem.
Cheers
On Fri, Apr 20, 2012 at 12:34 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Hi Jerry,
>
> which version of HBase are you using?
>
> You are not using cyclic backup, that needs >2 clusters. I assume you're
> just replicating from one cluster to another, right?
>
> There is never data loss in Cluster A?
>
> -- Lars
>
>
> ----- Original Message -----
> From: Jerry Lam <ch...@gmail.com>
> To: user@hbase.apache.org
> Cc:
> Sent: Friday, April 20, 2012 5:38 AM
> Subject: HBase Cyclic Replication Issue: some data are missing in the
> replication for intensive write
>
> Hi HBase community:
>
> We have been testing cyclic replication for 1 week. The basic
> functionality seems to work as described in the document however when we
> started to increase the write workload, the replication starts to miss data
> (i.e. some data are not replicated to the other cluster). We have narrowed
> down to a scenario that we can reproduce the problem quite consistently and
> here it is:
>
> -----------------------------
> Setup:
> - We have setup 2 clusters (cluster A and cluster B)with identical size in
> terms of number of nodes and configuration, 3 regionservers sit on top of 3
> datanodes.
> - Cyclic replication is enabled.
>
> - We use YCSB to generate load to hbase the workload is very similar to
> workloada:
>
> recordcount=200000
> operationcount=200000
> workload=com.yahoo.ycsb.workloads.CoreWorkload
> fieldcount=1
> fieldlength=25000
>
> readallfields=true
> writeallfields=true
>
> readproportion=0
> updateproportion=1
> scanproportion=0
> insertproportion=0
>
> requestdistribution=uniform
>
> - Records are inserted into Cluster A. After the benchmark is done and
> wait until all data are replicated to Cluster B, we used verifyrep
> mapreduce job for validation.
> - Data are deleted from both table (truncate 'tablename') before a new
> experiment is started.
>
> Scenario:
> when we increase the number of threads until it max out the throughput of
> the cluster, we saw some data are missing in Cluster B (total count !=
> 200000) although cluster A clearly has them all. This happens even though
> we disabled region splitting in both clusters (it happens more often when
> region splits occur). To further having more control of what is happening,
> we then decided to disable the load balancer so the region (which is
> responsible for the replicating data) will not relocate to other
> regionserver during the benchmark. The situation improves a lot. We don't
> see any missing data in 5 continuous runs. Finally, we decided to move the
> region around from a regionserver to another regionserver during the
> benchmark to see if the problem will reappear and it did.
>
> We believe that the issue could be related to region splitting and load
> balancing during intensive write, the hbase replication strategy hasn't yet
> cover those corner cases.
>
> Can someone take a look of it and suggest some ways to workaround this?
>
> Thanks~
>
> Jerry
>
Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Posted by lars hofhansl <lh...@yahoo.com>.
Hi Jerry,
which version of HBase are you using?
You are not using cyclic backup, that needs >2 clusters. I assume you're just replicating from one cluster to another, right?
There is never data loss in Cluster A?
-- Lars
----- Original Message -----
From: Jerry Lam <ch...@gmail.com>
To: user@hbase.apache.org
Cc:
Sent: Friday, April 20, 2012 5:38 AM
Subject: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write
Hi HBase community:
We have been testing cyclic replication for 1 week. The basic functionality seems to work as described in the document however when we started to increase the write workload, the replication starts to miss data (i.e. some data are not replicated to the other cluster). We have narrowed down to a scenario that we can reproduce the problem quite consistently and here it is:
-----------------------------
Setup:
- We have setup 2 clusters (cluster A and cluster B)with identical size in terms of number of nodes and configuration, 3 regionservers sit on top of 3 datanodes.
- Cyclic replication is enabled.
- We use YCSB to generate load to hbase the workload is very similar to workloada:
recordcount=200000
operationcount=200000
workload=com.yahoo.ycsb.workloads.CoreWorkload
fieldcount=1
fieldlength=25000
readallfields=true
writeallfields=true
readproportion=0
updateproportion=1
scanproportion=0
insertproportion=0
requestdistribution=uniform
- Records are inserted into Cluster A. After the benchmark is done and wait until all data are replicated to Cluster B, we used verifyrep mapreduce job for validation.
- Data are deleted from both table (truncate 'tablename') before a new experiment is started.
Scenario:
when we increase the number of threads until it max out the throughput of the cluster, we saw some data are missing in Cluster B (total count != 200000) although cluster A clearly has them all. This happens even though we disabled region splitting in both clusters (it happens more often when region splits occur). To further having more control of what is happening, we then decided to disable the load balancer so the region (which is responsible for the replicating data) will not relocate to other regionserver during the benchmark. The situation improves a lot. We don't see any missing data in 5 continuous runs. Finally, we decided to move the region around from a regionserver to another regionserver during the benchmark to see if the problem will reappear and it did.
We believe that the issue could be related to region splitting and load balancing during intensive write, the hbase replication strategy hasn't yet cover those corner cases.
Can someone take a look of it and suggest some ways to workaround this?
Thanks~
Jerry