You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Jean T. Anderson" <jt...@bristowhill.com> on 2012/02/18 01:24:16 UTC
HBase 0.90.4: YCSB gets NPE with large loads when timeseries is set
Hi, everyone,
I often get an NPE during the YCSB load process if timeseries is set and I'm loading lots of records. Here's the syntax I'm using:
YCSB Client 0.1
Command line: -load -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=family -p recordcount=10000000 -threads 32 -s -p measurementtype=timeseries -p timeseries.granularity=5000
100 million records load successfully with 32 threads if timeseries is not set. But when I set timeseries, I often get NPEs with stacks that move around depending on the number of records and the timeseries granularity.
For example:
1) 10 million records gets an NPE when timeseries is enabled with a granularity of 5000 ms (but 4000 ms loads successfully):
java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
at org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
at org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
at java.util.TreeMap$NavigableSubMap.checkUpperBound(TreeMap.java:2666)
at java.util.TreeMap$NavigableSubMap.isInRange(TreeMap.java:2688)
at java.util.TreeMap$NavigableSubMap.get(TreeMap.java:2502)
at org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:96)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:826)
2) 100 million records gets an NPE with 4000 ms (note the changed stack):
java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
at org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
at org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
at java.util.TreeMap$NavigableSubMap.cmp(TreeMap.java:2748)
at java.util.TreeMap$NavigableSubMap.findLowerEntryImpl(TreeMap.java:2881)
at java.util.TreeMap$NavigableSubMap.findLowerEntry(TreeMap.java:3000)
at java.util.TreeMap$NavigableSubMap.smallerEntry(TreeMap.java:2994)
at java.util.TreeMap$AscendingSubMapIterator.getBoundaryNode(TreeMap.java:281)
at java.util.TreeMap$AbstractSubMapIterator.<init>(TreeMap.java:186)
at java.util.TreeMap$AscendingSubMapKeySet.iterator(TreeMap.java:1968)
at java.util.TreeMap$NavigableSubMap.isEmpty(TreeMap.java:2477)
at org.apache.hadoop.hbase.util.SoftValueSortedMap.isEmpty(SoftValueSortedMap.java:151)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:826)
3) 100 million records gets an NPE with 3000 ms (and the stack changes):
java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
at org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
at org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
at java.util.TreeMap.cmp(TreeMap.java:4514)
at java.util.TreeMap.get(TreeMap.java:4386)
at org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:96)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:830)
I found HBASE-3878, but it looks like that was fixed in 0.90.4.
I also found HBASE-5088, which is marked fixed in 0.90.6 -- does anyone recognize if this problem fits that profile?
many thanks,
-jean
Re: HBase 0.90.4: YCSB gets NPE with large loads when timeseries is set
Posted by Stack <st...@duboce.net>.
On Mon, Feb 20, 2012 at 6:32 PM, pig user <pi...@gmail.com> wrote:
> It looks like HBASE-2909, but it was fixed in 0.90.0.
>
Do you have HBASE-2909 in your hbase and you are still seeing the
issue? Sounds like it wasn't fixed then?
St.Ack
Re: HBase 0.90.4: YCSB gets NPE with large loads when timeseries is set
Posted by pig user <pi...@gmail.com>.
It looks like HBASE-2909, but it was fixed in 0.90.0.
On Sat, Feb 18, 2012 at 6:48 PM, Jean T. Anderson <jt...@bristowhill.com>wrote:
> On 2/18/12 12:02 PM, Stack wrote:
>
>> What version are you running?
>>
> HBase 0.90.4 / hadoop 0.20.2
>
>
> I don't see any explicit NPE mentions in the commit log on
>> HConnectionManager. Maybe its still an issue?
>>
>> St.Ack
>>
>
> ok, thanks for the sanity check. I'll dig a little deeper and also see if
> the problem reproduces on 0.92.
>
> regards,
>
> -jean
>
>
> On Fri, Feb 17, 2012 at 4:24 PM, Jean T. Anderson<jt...@bristowhill.com>
>> wrote:
>>
>>> Hi, everyone,
>>>
>>> I often get an NPE during the YCSB load process if timeseries is set and
>>> I'm
>>> loading lots of records. Here's the syntax I'm using:
>>>
>>> YCSB Client 0.1
>>> Command line: -load -db com.yahoo.ycsb.db.HBaseClient -P
>>> workloads/workloada -p columnfamily=family -p recordcount=10000000
>>> -threads
>>> 32 -s -p measurementtype=timeseries -p timeseries.granularity=5000
>>>
>>> 100 million records load successfully with 32 threads if timeseries is
>>> not
>>> set. But when I set timeseries, I often get NPEs with stacks that move
>>> around depending on the number of records and the timeseries granularity.
>>>
>>> For example:
>>>
>>> 1) 10 million records gets an NPE when timeseries is enabled with a
>>> granularity of 5000 ms (but 4000 ms loads successfully):
>>>
>>> java.lang.NullPointerException
>>> at org.apache.hadoop.hbase.util.**Bytes.compareTo(Bytes.java:**
>>> 916)
>>> at
>>> org.apache.hadoop.hbase.util.**Bytes$ByteArrayComparator.**
>>> compare(Bytes.java:109)
>>> at
>>> org.apache.hadoop.hbase.util.**Bytes$ByteArrayComparator.**
>>> compare(Bytes.java:101)
>>> at
>>> java.util.TreeMap$**NavigableSubMap.**checkUpperBound(TreeMap.java:**
>>> 2666)
>>> at java.util.TreeMap$**NavigableSubMap.isInRange(**
>>> TreeMap.java:2688)
>>> at java.util.TreeMap$**NavigableSubMap.get(TreeMap.**java:2502)
>>> at
>>> org.apache.hadoop.hbase.util.**SoftValueSortedMap.get(**
>>> SoftValueSortedMap.java:96)
>>> at
>>> org.apache.hadoop.hbase.**client.HConnectionManager$**
>>> HConnectionImplementation.**getCachedLocation(**
>>> HConnectionManager.java:826)
>>>
>>> 2) 100 million records gets an NPE with 4000 ms (note the changed stack):
>>>
>>> java.lang.NullPointerException
>>> at org.apache.hadoop.hbase.util.**Bytes.compareTo(Bytes.java:**
>>> 916)
>>> at
>>> org.apache.hadoop.hbase.util.**Bytes$ByteArrayComparator.**
>>> compare(Bytes.java:109)
>>> at
>>> org.apache.hadoop.hbase.util.**Bytes$ByteArrayComparator.**
>>> compare(Bytes.java:101)
>>> at java.util.TreeMap$**NavigableSubMap.cmp(TreeMap.**java:2748)
>>> at
>>> java.util.TreeMap$**NavigableSubMap.**findLowerEntryImpl(TreeMap.**
>>> java:2881)
>>> at
>>> java.util.TreeMap$**NavigableSubMap.**findLowerEntry(TreeMap.java:**
>>> 3000)
>>> at java.util.TreeMap$**NavigableSubMap.smallerEntry(**
>>> TreeMap.java:2994)
>>> at
>>> java.util.TreeMap$**AscendingSubMapIterator.**
>>> getBoundaryNode(TreeMap.java:**281)
>>> at java.util.TreeMap$**AbstractSubMapIterator.<init>(**
>>> TreeMap.java:186)
>>> at
>>> java.util.TreeMap$**AscendingSubMapKeySet.**iterator(TreeMap.java:1968)
>>> at java.util.TreeMap$**NavigableSubMap.isEmpty(**
>>> TreeMap.java:2477)
>>> at
>>> org.apache.hadoop.hbase.util.**SoftValueSortedMap.isEmpty(**
>>> SoftValueSortedMap.java:151)
>>> at
>>> org.apache.hadoop.hbase.**client.HConnectionManager$**
>>> HConnectionImplementation.**getCachedLocation(**
>>> HConnectionManager.java:826)
>>>
>>> 3) 100 million records gets an NPE with 3000 ms (and the stack changes):
>>>
>>> java.lang.NullPointerException
>>> at org.apache.hadoop.hbase.util.**Bytes.compareTo(Bytes.java:**
>>> 916)
>>> at
>>> org.apache.hadoop.hbase.util.**Bytes$ByteArrayComparator.**
>>> compare(Bytes.java:109)
>>> at
>>> org.apache.hadoop.hbase.util.**Bytes$ByteArrayComparator.**
>>> compare(Bytes.java:101)
>>> at java.util.TreeMap.cmp(TreeMap.**java:4514)
>>> at java.util.TreeMap.get(TreeMap.**java:4386)
>>> at
>>> org.apache.hadoop.hbase.util.**SoftValueSortedMap.get(**
>>> SoftValueSortedMap.java:96)
>>> at
>>> org.apache.hadoop.hbase.**client.HConnectionManager$**
>>> HConnectionImplementation.**getCachedLocation(**
>>> HConnectionManager.java:830)
>>>
>>> I found HBASE-3878, but it looks like that was fixed in 0.90.4.
>>> I also found HBASE-5088, which is marked fixed in 0.90.6 -- does anyone
>>> recognize if this problem fits that profile?
>>>
>>> many thanks,
>>>
>>> -jean
>>>
>>>
>>>
>
Re: HBase 0.90.4: YCSB gets NPE with large loads when timeseries
is set
Posted by "Jean T. Anderson" <jt...@bristowhill.com>.
On 2/18/12 12:02 PM, Stack wrote:
> What version are you running?
HBase 0.90.4 / hadoop 0.20.2
> I don't see any explicit NPE mentions in the commit log on
> HConnectionManager. Maybe its still an issue?
>
> St.Ack
ok, thanks for the sanity check. I'll dig a little deeper and also see
if the problem reproduces on 0.92.
regards,
-jean
> On Fri, Feb 17, 2012 at 4:24 PM, Jean T. Anderson<jt...@bristowhill.com> wrote:
>> Hi, everyone,
>>
>> I often get an NPE during the YCSB load process if timeseries is set and I'm
>> loading lots of records. Here's the syntax I'm using:
>>
>> YCSB Client 0.1
>> Command line: -load -db com.yahoo.ycsb.db.HBaseClient -P
>> workloads/workloada -p columnfamily=family -p recordcount=10000000 -threads
>> 32 -s -p measurementtype=timeseries -p timeseries.granularity=5000
>>
>> 100 million records load successfully with 32 threads if timeseries is not
>> set. But when I set timeseries, I often get NPEs with stacks that move
>> around depending on the number of records and the timeseries granularity.
>>
>> For example:
>>
>> 1) 10 million records gets an NPE when timeseries is enabled with a
>> granularity of 5000 ms (but 4000 ms loads successfully):
>>
>> java.lang.NullPointerException
>> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
>> at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
>> at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
>> at
>> java.util.TreeMap$NavigableSubMap.checkUpperBound(TreeMap.java:2666)
>> at java.util.TreeMap$NavigableSubMap.isInRange(TreeMap.java:2688)
>> at java.util.TreeMap$NavigableSubMap.get(TreeMap.java:2502)
>> at
>> org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:96)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:826)
>>
>> 2) 100 million records gets an NPE with 4000 ms (note the changed stack):
>>
>> java.lang.NullPointerException
>> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
>> at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
>> at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
>> at java.util.TreeMap$NavigableSubMap.cmp(TreeMap.java:2748)
>> at
>> java.util.TreeMap$NavigableSubMap.findLowerEntryImpl(TreeMap.java:2881)
>> at
>> java.util.TreeMap$NavigableSubMap.findLowerEntry(TreeMap.java:3000)
>> at java.util.TreeMap$NavigableSubMap.smallerEntry(TreeMap.java:2994)
>> at
>> java.util.TreeMap$AscendingSubMapIterator.getBoundaryNode(TreeMap.java:281)
>> at java.util.TreeMap$AbstractSubMapIterator.<init>(TreeMap.java:186)
>> at
>> java.util.TreeMap$AscendingSubMapKeySet.iterator(TreeMap.java:1968)
>> at java.util.TreeMap$NavigableSubMap.isEmpty(TreeMap.java:2477)
>> at
>> org.apache.hadoop.hbase.util.SoftValueSortedMap.isEmpty(SoftValueSortedMap.java:151)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:826)
>>
>> 3) 100 million records gets an NPE with 3000 ms (and the stack changes):
>>
>> java.lang.NullPointerException
>> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
>> at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
>> at
>> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
>> at java.util.TreeMap.cmp(TreeMap.java:4514)
>> at java.util.TreeMap.get(TreeMap.java:4386)
>> at
>> org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:96)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:830)
>>
>> I found HBASE-3878, but it looks like that was fixed in 0.90.4.
>> I also found HBASE-5088, which is marked fixed in 0.90.6 -- does anyone
>> recognize if this problem fits that profile?
>>
>> many thanks,
>>
>> -jean
>>
>>
Re: HBase 0.90.4: YCSB gets NPE with large loads when timeseries is set
Posted by Stack <st...@duboce.net>.
What version are you running?
I don't see any explicit NPE mentions in the commit log on
HConnectionManager. Maybe its still an issue?
St.Ack
On Fri, Feb 17, 2012 at 4:24 PM, Jean T. Anderson <jt...@bristowhill.com> wrote:
> Hi, everyone,
>
> I often get an NPE during the YCSB load process if timeseries is set and I'm
> loading lots of records. Here's the syntax I'm using:
>
> YCSB Client 0.1
> Command line: -load -db com.yahoo.ycsb.db.HBaseClient -P
> workloads/workloada -p columnfamily=family -p recordcount=10000000 -threads
> 32 -s -p measurementtype=timeseries -p timeseries.granularity=5000
>
> 100 million records load successfully with 32 threads if timeseries is not
> set. But when I set timeseries, I often get NPEs with stacks that move
> around depending on the number of records and the timeseries granularity.
>
> For example:
>
> 1) 10 million records gets an NPE when timeseries is enabled with a
> granularity of 5000 ms (but 4000 ms loads successfully):
>
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
> at
> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
> at
> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
> at
> java.util.TreeMap$NavigableSubMap.checkUpperBound(TreeMap.java:2666)
> at java.util.TreeMap$NavigableSubMap.isInRange(TreeMap.java:2688)
> at java.util.TreeMap$NavigableSubMap.get(TreeMap.java:2502)
> at
> org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:96)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:826)
>
> 2) 100 million records gets an NPE with 4000 ms (note the changed stack):
>
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
> at
> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
> at
> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
> at java.util.TreeMap$NavigableSubMap.cmp(TreeMap.java:2748)
> at
> java.util.TreeMap$NavigableSubMap.findLowerEntryImpl(TreeMap.java:2881)
> at
> java.util.TreeMap$NavigableSubMap.findLowerEntry(TreeMap.java:3000)
> at java.util.TreeMap$NavigableSubMap.smallerEntry(TreeMap.java:2994)
> at
> java.util.TreeMap$AscendingSubMapIterator.getBoundaryNode(TreeMap.java:281)
> at java.util.TreeMap$AbstractSubMapIterator.<init>(TreeMap.java:186)
> at
> java.util.TreeMap$AscendingSubMapKeySet.iterator(TreeMap.java:1968)
> at java.util.TreeMap$NavigableSubMap.isEmpty(TreeMap.java:2477)
> at
> org.apache.hadoop.hbase.util.SoftValueSortedMap.isEmpty(SoftValueSortedMap.java:151)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:826)
>
> 3) 100 million records gets an NPE with 3000 ms (and the stack changes):
>
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:916)
> at
> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:109)
> at
> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:101)
> at java.util.TreeMap.cmp(TreeMap.java:4514)
> at java.util.TreeMap.get(TreeMap.java:4386)
> at
> org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:96)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:830)
>
> I found HBASE-3878, but it looks like that was fixed in 0.90.4.
> I also found HBASE-5088, which is marked fixed in 0.90.6 -- does anyone
> recognize if this problem fits that profile?
>
> many thanks,
>
> -jean
>
>