You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Oliver Meyn (GBIF)" <om...@gbif.org> on 2012/02/14 16:56:38 UTC
strange PerformanceEvaluation behaviour
Hi all,
I've been trying to run a battery of tests to really understand our cluster's performance, and I'm employing PerformanceEvaluation to do that (picking up where Tim Robertson left off, elsewhere on the list). I'm seeing two strange things that I hope someone can help with:
1) With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 10' I see 100 mappers spawned, rather than the expected 10. I expect 10 because that's what the usage text implies, and what the javadoc explicitly states - quoting from doMapReduce "Run as many maps as asked-for clients." The culprit appears to be the outer loop in writeInputFile which sets up 10 splits for every "asked-for client" - at least, if I'm reading it right. Is this somehow expected, or is that code leftover from some previous iteration/experiment?
2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B).
Any and all input appreciated.
Thanks,
Oliver
--
Oliver Meyn
Software Developer
Global Biodiversity Information Facility (GBIF)
+45 35 32 15 12
http://www.gbif.org
Re: strange PerformanceEvaluation behaviour
Posted by "Oliver Meyn (GBIF)" <om...@gbif.org>.
On 2012-02-15, at 5:39 PM, Stack wrote:
> On Wed, Feb 15, 2012 at 1:53 AM, Oliver Meyn (GBIF) <om...@gbif.org> wrote:
>> So hacking around reveals that key collision is indeed the problem. I thought the modulo part of the getRandomRow method was suspect but while removing it improved the behaviour (I got ~8M rows instead of ~6.6M) it didn't fix it completely. Since that's really what UUIDs are for I gave that a shot (i.e UUID.randomUUID()) and sure enough now I get the full 10M rows. Those are 16-byte keys now though, instead of the 10-byte that the integers produced. But because we're testing scan performance I think using a sequentially written table would probably be cheating and so will stick with randomWrite with slightly bigger keys. That means it's a little harder to compare to the results that other people get, but at least I know my internal tests are apples to apples.
>>
>> Oh and I removed the outer 10x loop and that produced the desired number of mappers (ie what I passed in on the commandline) but made no difference in the key generation/collision story.
>>
>> Should I file bugs for these 2 issues?
>>
>
> Thanks Oliver for digging.
>
> Using UUIDs will make it tougher on the other end when reading? How
> do you divide up the UUID space? UUIDs are not well distributed
> across the possible key space IIUC.
>
> Should writing UUIDs be an option on PE?
>
> Thanks again for figuring it.
>
> St.Ack
>
Honestly I don't know very much about UUIDs so I didn't consider their distribution over the keyspace - just used UUID.randomUUID() and more or less crossed my fingers :) I agree that they're a bit of a PITA when it comes to reading them, but I think having exactly the expected number of rows written and read in the test makes the PE more obvious and therefore more useful/useable. So yes, I think UUIDs as a key option in PE would be good. I left some code in the JIRA as a starting point for a patch.
Cheers,
Oliver
Re: strange PerformanceEvaluation behaviour
Posted by Stack <st...@duboce.net>.
On Wed, Feb 15, 2012 at 1:53 AM, Oliver Meyn (GBIF) <om...@gbif.org> wrote:
> So hacking around reveals that key collision is indeed the problem. I thought the modulo part of the getRandomRow method was suspect but while removing it improved the behaviour (I got ~8M rows instead of ~6.6M) it didn't fix it completely. Since that's really what UUIDs are for I gave that a shot (i.e UUID.randomUUID()) and sure enough now I get the full 10M rows. Those are 16-byte keys now though, instead of the 10-byte that the integers produced. But because we're testing scan performance I think using a sequentially written table would probably be cheating and so will stick with randomWrite with slightly bigger keys. That means it's a little harder to compare to the results that other people get, but at least I know my internal tests are apples to apples.
>
> Oh and I removed the outer 10x loop and that produced the desired number of mappers (ie what I passed in on the commandline) but made no difference in the key generation/collision story.
>
> Should I file bugs for these 2 issues?
>
Thanks Oliver for digging.
Using UUIDs will make it tougher on the other end when reading? How
do you divide up the UUID space? UUIDs are not well distributed
across the possible key space IIUC.
Should writing UUIDs be an option on PE?
Thanks again for figuring it.
St.Ack
Re: strange PerformanceEvaluation behaviour
Posted by "Oliver Meyn (GBIF)" <om...@gbif.org>.
Okie:
10x # of mappers: https://issues.apache.org/jira/browse/HBASE-5401
wrong row count: https://issues.apache.org/jira/browse/HBASE-5402
Oliver
On 2012-02-15, at 11:50 AM, yuzhihong@gmail.com wrote:
> Oliver:
> Thanks for digging.
>
> Please file Jira's for these issues.
>
>
>
> On Feb 15, 2012, at 1:53 AM, "Oliver Meyn (GBIF)" <om...@gbif.org> wrote:
>
>> On 2012-02-15, at 9:09 AM, Oliver Meyn (GBIF) wrote:
>>
>>> On 2012-02-15, at 7:32 AM, Stack wrote:
>>>
>>>> On Tue, Feb 14, 2012 at 8:14 AM, Stack <st...@duboce.net> wrote:
>>>>>> 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B).
>>>>>>
>>>>>
>>>>
>>>> I just tried it here and got similar result. I wonder if its the
>>>> randomWrite? What if you do sequentialWrite, do you get our 10M?
>>>
>>> Thanks for checking into this stack - when using sequentialWrite I get the expected 10485700 rows. I'll hack around a bit on the PE to count the number of collisions, and try to think of a reasonable solution.
>>
>> So hacking around reveals that key collision is indeed the problem. I thought the modulo part of the getRandomRow method was suspect but while removing it improved the behaviour (I got ~8M rows instead of ~6.6M) it didn't fix it completely. Since that's really what UUIDs are for I gave that a shot (i.e UUID.randomUUID()) and sure enough now I get the full 10M rows. Those are 16-byte keys now though, instead of the 10-byte that the integers produced. But because we're testing scan performance I think using a sequentially written table would probably be cheating and so will stick with randomWrite with slightly bigger keys. That means it's a little harder to compare to the results that other people get, but at least I know my internal tests are apples to apples.
>>
>> Oh and I removed the outer 10x loop and that produced the desired number of mappers (ie what I passed in on the commandline) but made no difference in the key generation/collision story.
>>
>> Should I file bugs for these 2 issues?
>>
>> Thanks,
>> Oliver
>>
Re: strange PerformanceEvaluation behaviour
Posted by yu...@gmail.com.
Oliver:
Thanks for digging.
Please file Jira's for these issues.
On Feb 15, 2012, at 1:53 AM, "Oliver Meyn (GBIF)" <om...@gbif.org> wrote:
> On 2012-02-15, at 9:09 AM, Oliver Meyn (GBIF) wrote:
>
>> On 2012-02-15, at 7:32 AM, Stack wrote:
>>
>>> On Tue, Feb 14, 2012 at 8:14 AM, Stack <st...@duboce.net> wrote:
>>>>> 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B).
>>>>>
>>>>
>>>
>>> I just tried it here and got similar result. I wonder if its the
>>> randomWrite? What if you do sequentialWrite, do you get our 10M?
>>
>> Thanks for checking into this stack - when using sequentialWrite I get the expected 10485700 rows. I'll hack around a bit on the PE to count the number of collisions, and try to think of a reasonable solution.
>
> So hacking around reveals that key collision is indeed the problem. I thought the modulo part of the getRandomRow method was suspect but while removing it improved the behaviour (I got ~8M rows instead of ~6.6M) it didn't fix it completely. Since that's really what UUIDs are for I gave that a shot (i.e UUID.randomUUID()) and sure enough now I get the full 10M rows. Those are 16-byte keys now though, instead of the 10-byte that the integers produced. But because we're testing scan performance I think using a sequentially written table would probably be cheating and so will stick with randomWrite with slightly bigger keys. That means it's a little harder to compare to the results that other people get, but at least I know my internal tests are apples to apples.
>
> Oh and I removed the outer 10x loop and that produced the desired number of mappers (ie what I passed in on the commandline) but made no difference in the key generation/collision story.
>
> Should I file bugs for these 2 issues?
>
> Thanks,
> Oliver
>
Re: strange PerformanceEvaluation behaviour
Posted by "Oliver Meyn (GBIF)" <om...@gbif.org>.
On 2012-02-15, at 9:09 AM, Oliver Meyn (GBIF) wrote:
> On 2012-02-15, at 7:32 AM, Stack wrote:
>
>> On Tue, Feb 14, 2012 at 8:14 AM, Stack <st...@duboce.net> wrote:
>>>> 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B).
>>>>
>>>
>>
>> I just tried it here and got similar result. I wonder if its the
>> randomWrite? What if you do sequentialWrite, do you get our 10M?
>
> Thanks for checking into this stack - when using sequentialWrite I get the expected 10485700 rows. I'll hack around a bit on the PE to count the number of collisions, and try to think of a reasonable solution.
So hacking around reveals that key collision is indeed the problem. I thought the modulo part of the getRandomRow method was suspect but while removing it improved the behaviour (I got ~8M rows instead of ~6.6M) it didn't fix it completely. Since that's really what UUIDs are for I gave that a shot (i.e UUID.randomUUID()) and sure enough now I get the full 10M rows. Those are 16-byte keys now though, instead of the 10-byte that the integers produced. But because we're testing scan performance I think using a sequentially written table would probably be cheating and so will stick with randomWrite with slightly bigger keys. That means it's a little harder to compare to the results that other people get, but at least I know my internal tests are apples to apples.
Oh and I removed the outer 10x loop and that produced the desired number of mappers (ie what I passed in on the commandline) but made no difference in the key generation/collision story.
Should I file bugs for these 2 issues?
Thanks,
Oliver
Re: strange PerformanceEvaluation behaviour
Posted by "Oliver Meyn (GBIF)" <om...@gbif.org>.
On 2012-02-15, at 7:32 AM, Stack wrote:
> On Tue, Feb 14, 2012 at 8:14 AM, Stack <st...@duboce.net> wrote:
>>> 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B).
>>>
>>
>
> I just tried it here and got similar result. I wonder if its the
> randomWrite? What if you do sequentialWrite, do you get our 10M?
Thanks for checking into this stack - when using sequentialWrite I get the expected 10485700 rows. I'll hack around a bit on the PE to count the number of collisions, and try to think of a reasonable solution.
Thanks,
Oliver
Re: strange PerformanceEvaluation behaviour
Posted by Stack <st...@duboce.net>.
On Tue, Feb 14, 2012 at 8:14 AM, Stack <st...@duboce.net> wrote:
>> 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B).
>>
>
I just tried it here and got similar result. I wonder if its the
randomWrite? What if you do sequentialWrite, do you get our 10M?
I'll try it here again later with sequentialWrite.
St.Ack
Re: strange PerformanceEvaluation behaviour
Posted by Stack <st...@duboce.net>.
On Tue, Feb 14, 2012 at 7:56 AM, Oliver Meyn (GBIF) <om...@gbif.org> wrote:
> 1) With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 10' I see 100 mappers spawned, rather than the expected 10. I expect 10 because that's what the usage text implies, and what the javadoc explicitly states - quoting from doMapReduce "Run as many maps as asked-for clients." The culprit appears to be the outer loop in writeInputFile which sets up 10 splits for every "asked-for client" - at least, if I'm reading it right. Is this somehow expected, or is that code leftover from some previous iteration/experiment?
>
Yeah. I'd expect ten clients, each to its own map, each doing 1M items each.
Looking at writeInputFile, it seems to be dividing the namespace by
ten so, yeah x10 mappers.
> 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B).
>
Yeah, I'd think key overlap (print out the span for each mappers or
check that file written by writeInputFile).
Your clocks are all in sync?
St.Ack