You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Gaojinchao <ga...@huawei.com> on 2011/05/09 14:22:25 UTC
A question about client
I used ycsb to put data and threw exception.
Who can give me some suggestion?
Hbase Code:
// Cut the cache so that we only get the part that could contain
// regions that match our key
SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
tableLocations.headMap(row);
// if that portion of the map is empty, then we're done. otherwise,
// we need to examine the cached location to verify that it is
// a match by end key as well.
if (!matchingRegions.isEmpty()) {
HRegionLocation possibleRegion =
matchingRegions.get(matchingRegions.lastKey());
ycsb client log:
[java] begin StatusThread run
[java] java.util.NoSuchElementException
[java] at java.util.TreeMap.key(TreeMap.java:1206)
[java] at java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
[java] at org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
[java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
[java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
[java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
[java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
[java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
[java] at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
[java] at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
[java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
[java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
[java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
[java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
[java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown Source)
[java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
re: A question about client
Posted by Gaojinchao <ga...@huawei.com>.
Make a issue:
https://issues.apache.org/jira/browse/HBASE-3878
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2011年5月12日 17:52
收件人: user@hbase.apache.org
抄送: user@hbase.apache.org
主题: Re: A question about client
Yes please.
On May 12, 2011, at 12:49 AM, Gaojinchao <ga...@huawei.com> wrote:
> Does it need open a issue ?
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2011年5月11日 10:31
> 收件人: user@hbase.apache.org
> 主题: Re: A question about client
>
> I think the second explanation is plausible.
> From
> http://download.oracle.com/javase/1.4.2/docs/api/java/lang/ref/SoftReference.html
> :
> Soft reference objects, which are cleared at the discretion of the garbage
> collector in response to memory demand.
>
> I think we should check whether possibleRegion is null in the code below.
>
> 2011/5/10 Jean-Daniel Cryans <jd...@apache.org>
>
>> Are you running a modified YCSB by any chance? Because last time I
>> looked at that code it didn't share the HTables between threads and it
>> looks like it's doing something like that.
>>
>> Looking more deeper at the code, the NoSuchElementException is thrown
>> because the map is empty. This is what that code looks like:
>>
>> if (!matchingRegions.isEmpty()) {
>> HRegionLocation possibleRegion =
>> matchingRegions.get(matchingRegions.lastKey());
>>
>> So to me it seems that the only way you would get this exception is if
>> someone emptied the map between the isEmpty call and lastKey which
>> shouldn't happen if HTables aren't shared.
>>
>> The only other way it seems it could happen, and it's a stretch, is
>> that since the regions are kept in a SoftValueSortedMap then the GC
>> would have removed the elements you needed exactly between those two
>> lines... Is it easy for you to recreate the issue?
>>
>> Thx a bunch,
>>
>> J-D
>>
>> On Mon, May 9, 2011 at 11:34 PM, Gaojinchao <ga...@huawei.com> wrote:
>>> Hbase version: 0.90.2 .
>>> I merged patches:
>>> HBASE-3773 Set ZK max connections much higher in 0.90
>>> HBASE-3771 All jsp pages don't clean their HBA
>>> HBASE-3783 hbase-0.90.2.jar exists in hbase root and in 'lib/'
>>> HBASE-3756 Can't move META or ROOT from shell
>>> HBASE-3744 createTable blocks until all regions are out of
>> transition
>>> HBASE-3712 HTable.close() doesn't shutdown thread pool
>>> HBASE-3750 HTablePool.putTable() should call
>> tableFactory.releaseHTableInterface() for discarded table
>>> HBASE-3722 A lot of data is lost when name node crashed
>>> HBASE-3800 If HMaster is started after NN without starting DN in
>> Hbase 090.2 then HMaster is not able to start due to AlreadyCreatedException
>> for /hbase/hbase.version
>>> HBASE-3749 Master can't exit when open port failed
>>>
>>> -----邮件原件-----
>>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel
>> Cryans
>>> 发送时间: 2011年5月10日 1:17
>>> 收件人: user@hbase.apache.org
>>> 主题: Re: A question about client
>>>
>>> TreeMap isn't concurrent and it seems it was used that way? I know you
>>> guys are testing a bunch of different things at the same time so which
>>> HBase version and which patches were you using when you got that?
>>>
>>> Thx,
>>>
>>> J-D
>>>
>>> On Mon, May 9, 2011 at 5:22 AM, Gaojinchao <ga...@huawei.com>
>> wrote:
>>>> I used ycsb to put data and threw exception.
>>>> Who can give me some suggestion?
>>>>
>>>> Hbase Code:
>>>> // Cut the cache so that we only get the part that could contain
>>>> // regions that match our key
>>>> SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
>>>> tableLocations.headMap(row);
>>>>
>>>> // if that portion of the map is empty, then we're done. otherwise,
>>>> // we need to examine the cached location to verify that it is
>>>> // a match by end key as well.
>>>> if (!matchingRegions.isEmpty()) {
>>>> HRegionLocation possibleRegion =
>>>> matchingRegions.get(matchingRegions.lastKey());
>>>>
>>>> ycsb client log:
>>>>
>>>> [java] begin StatusThread run
>>>> [java] java.util.NoSuchElementException
>>>> [java] at java.util.TreeMap.key(TreeMap.java:1206)
>>>> [java] at
>> java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
>>>> [java] at
>> org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
>>>> [java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
>>>> [java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
>>>> [java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
>>>> [java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown
>> Source)
>>>> [java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
>>>>
>>>
>>
Re: A question about client
Posted by Ted Yu <yu...@gmail.com>.
Yes please.
On May 12, 2011, at 12:49 AM, Gaojinchao <ga...@huawei.com> wrote:
> Does it need open a issue ?
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2011年5月11日 10:31
> 收件人: user@hbase.apache.org
> 主题: Re: A question about client
>
> I think the second explanation is plausible.
> From
> http://download.oracle.com/javase/1.4.2/docs/api/java/lang/ref/SoftReference.html
> :
> Soft reference objects, which are cleared at the discretion of the garbage
> collector in response to memory demand.
>
> I think we should check whether possibleRegion is null in the code below.
>
> 2011/5/10 Jean-Daniel Cryans <jd...@apache.org>
>
>> Are you running a modified YCSB by any chance? Because last time I
>> looked at that code it didn't share the HTables between threads and it
>> looks like it's doing something like that.
>>
>> Looking more deeper at the code, the NoSuchElementException is thrown
>> because the map is empty. This is what that code looks like:
>>
>> if (!matchingRegions.isEmpty()) {
>> HRegionLocation possibleRegion =
>> matchingRegions.get(matchingRegions.lastKey());
>>
>> So to me it seems that the only way you would get this exception is if
>> someone emptied the map between the isEmpty call and lastKey which
>> shouldn't happen if HTables aren't shared.
>>
>> The only other way it seems it could happen, and it's a stretch, is
>> that since the regions are kept in a SoftValueSortedMap then the GC
>> would have removed the elements you needed exactly between those two
>> lines... Is it easy for you to recreate the issue?
>>
>> Thx a bunch,
>>
>> J-D
>>
>> On Mon, May 9, 2011 at 11:34 PM, Gaojinchao <ga...@huawei.com> wrote:
>>> Hbase version: 0.90.2 .
>>> I merged patches:
>>> HBASE-3773 Set ZK max connections much higher in 0.90
>>> HBASE-3771 All jsp pages don't clean their HBA
>>> HBASE-3783 hbase-0.90.2.jar exists in hbase root and in 'lib/'
>>> HBASE-3756 Can't move META or ROOT from shell
>>> HBASE-3744 createTable blocks until all regions are out of
>> transition
>>> HBASE-3712 HTable.close() doesn't shutdown thread pool
>>> HBASE-3750 HTablePool.putTable() should call
>> tableFactory.releaseHTableInterface() for discarded table
>>> HBASE-3722 A lot of data is lost when name node crashed
>>> HBASE-3800 If HMaster is started after NN without starting DN in
>> Hbase 090.2 then HMaster is not able to start due to AlreadyCreatedException
>> for /hbase/hbase.version
>>> HBASE-3749 Master can't exit when open port failed
>>>
>>> -----邮件原件-----
>>> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel
>> Cryans
>>> 发送时间: 2011年5月10日 1:17
>>> 收件人: user@hbase.apache.org
>>> 主题: Re: A question about client
>>>
>>> TreeMap isn't concurrent and it seems it was used that way? I know you
>>> guys are testing a bunch of different things at the same time so which
>>> HBase version and which patches were you using when you got that?
>>>
>>> Thx,
>>>
>>> J-D
>>>
>>> On Mon, May 9, 2011 at 5:22 AM, Gaojinchao <ga...@huawei.com>
>> wrote:
>>>> I used ycsb to put data and threw exception.
>>>> Who can give me some suggestion?
>>>>
>>>> Hbase Code:
>>>> // Cut the cache so that we only get the part that could contain
>>>> // regions that match our key
>>>> SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
>>>> tableLocations.headMap(row);
>>>>
>>>> // if that portion of the map is empty, then we're done. otherwise,
>>>> // we need to examine the cached location to verify that it is
>>>> // a match by end key as well.
>>>> if (!matchingRegions.isEmpty()) {
>>>> HRegionLocation possibleRegion =
>>>> matchingRegions.get(matchingRegions.lastKey());
>>>>
>>>> ycsb client log:
>>>>
>>>> [java] begin StatusThread run
>>>> [java] java.util.NoSuchElementException
>>>> [java] at java.util.TreeMap.key(TreeMap.java:1206)
>>>> [java] at
>> java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
>>>> [java] at
>> org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
>>>> [java] at
>> org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
>>>> [java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
>>>> [java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
>>>> [java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
>>>> [java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown
>> Source)
>>>> [java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
>>>>
>>>
>>
re: A question about client
Posted by Gaojinchao <ga...@huawei.com>.
Does it need open a issue ?
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2011年5月11日 10:31
收件人: user@hbase.apache.org
主题: Re: A question about client
I think the second explanation is plausible.
From
http://download.oracle.com/javase/1.4.2/docs/api/java/lang/ref/SoftReference.html
:
Soft reference objects, which are cleared at the discretion of the garbage
collector in response to memory demand.
I think we should check whether possibleRegion is null in the code below.
2011/5/10 Jean-Daniel Cryans <jd...@apache.org>
> Are you running a modified YCSB by any chance? Because last time I
> looked at that code it didn't share the HTables between threads and it
> looks like it's doing something like that.
>
> Looking more deeper at the code, the NoSuchElementException is thrown
> because the map is empty. This is what that code looks like:
>
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
>
> So to me it seems that the only way you would get this exception is if
> someone emptied the map between the isEmpty call and lastKey which
> shouldn't happen if HTables aren't shared.
>
> The only other way it seems it could happen, and it's a stretch, is
> that since the regions are kept in a SoftValueSortedMap then the GC
> would have removed the elements you needed exactly between those two
> lines... Is it easy for you to recreate the issue?
>
> Thx a bunch,
>
> J-D
>
> On Mon, May 9, 2011 at 11:34 PM, Gaojinchao <ga...@huawei.com> wrote:
> > Hbase version: 0.90.2 .
> > I merged patches:
> > HBASE-3773 Set ZK max connections much higher in 0.90
> > HBASE-3771 All jsp pages don't clean their HBA
> > HBASE-3783 hbase-0.90.2.jar exists in hbase root and in 'lib/'
> > HBASE-3756 Can't move META or ROOT from shell
> > HBASE-3744 createTable blocks until all regions are out of
> transition
> > HBASE-3712 HTable.close() doesn't shutdown thread pool
> > HBASE-3750 HTablePool.putTable() should call
> tableFactory.releaseHTableInterface() for discarded table
> > HBASE-3722 A lot of data is lost when name node crashed
> > HBASE-3800 If HMaster is started after NN without starting DN in
> Hbase 090.2 then HMaster is not able to start due to AlreadyCreatedException
> for /hbase/hbase.version
> > HBASE-3749 Master can't exit when open port failed
> >
> > -----邮件原件-----
> > 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel
> Cryans
> > 发送时间: 2011年5月10日 1:17
> > 收件人: user@hbase.apache.org
> > 主题: Re: A question about client
> >
> > TreeMap isn't concurrent and it seems it was used that way? I know you
> > guys are testing a bunch of different things at the same time so which
> > HBase version and which patches were you using when you got that?
> >
> > Thx,
> >
> > J-D
> >
> > On Mon, May 9, 2011 at 5:22 AM, Gaojinchao <ga...@huawei.com>
> wrote:
> >> I used ycsb to put data and threw exception.
> >> Who can give me some suggestion?
> >>
> >> Hbase Code:
> >> // Cut the cache so that we only get the part that could contain
> >> // regions that match our key
> >> SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
> >> tableLocations.headMap(row);
> >>
> >> // if that portion of the map is empty, then we're done. otherwise,
> >> // we need to examine the cached location to verify that it is
> >> // a match by end key as well.
> >> if (!matchingRegions.isEmpty()) {
> >> HRegionLocation possibleRegion =
> >> matchingRegions.get(matchingRegions.lastKey());
> >>
> >> ycsb client log:
> >>
> >> [java] begin StatusThread run
> >> [java] java.util.NoSuchElementException
> >> [java] at java.util.TreeMap.key(TreeMap.java:1206)
> >> [java] at
> java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
> >> [java] at
> org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
> >> [java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
> >> [java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
> >> [java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
> >> [java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown
> Source)
> >> [java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
> >>
> >
>
Re: A question about client
Posted by Gaojinchao <ga...@huawei.com>.
Thanks. I think that gc resulted in
It is difficult to reproduce. I find it firstly in performance test cluster.
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2011年5月11日 10:31
收件人: user@hbase.apache.org
主题: Re: A question about client
I think the second explanation is plausible.
From
http://download.oracle.com/javase/1.4.2/docs/api/java/lang/ref/SoftReference.html
:
Soft reference objects, which are cleared at the discretion of the garbage
collector in response to memory demand.
I think we should check whether possibleRegion is null in the code below.
2011/5/10 Jean-Daniel Cryans <jd...@apache.org>
> Are you running a modified YCSB by any chance? Because last time I
> looked at that code it didn't share the HTables between threads and it
> looks like it's doing something like that.
>
> Looking more deeper at the code, the NoSuchElementException is thrown
> because the map is empty. This is what that code looks like:
>
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
>
> So to me it seems that the only way you would get this exception is if
> someone emptied the map between the isEmpty call and lastKey which
> shouldn't happen if HTables aren't shared.
>
> The only other way it seems it could happen, and it's a stretch, is
> that since the regions are kept in a SoftValueSortedMap then the GC
> would have removed the elements you needed exactly between those two
> lines... Is it easy for you to recreate the issue?
>
> Thx a bunch,
>
> J-D
>
> On Mon, May 9, 2011 at 11:34 PM, Gaojinchao <ga...@huawei.com> wrote:
> > Hbase version: 0.90.2 .
> > I merged patches:
> > HBASE-3773 Set ZK max connections much higher in 0.90
> > HBASE-3771 All jsp pages don't clean their HBA
> > HBASE-3783 hbase-0.90.2.jar exists in hbase root and in 'lib/'
> > HBASE-3756 Can't move META or ROOT from shell
> > HBASE-3744 createTable blocks until all regions are out of
> transition
> > HBASE-3712 HTable.close() doesn't shutdown thread pool
> > HBASE-3750 HTablePool.putTable() should call
> tableFactory.releaseHTableInterface() for discarded table
> > HBASE-3722 A lot of data is lost when name node crashed
> > HBASE-3800 If HMaster is started after NN without starting DN in
> Hbase 090.2 then HMaster is not able to start due to AlreadyCreatedException
> for /hbase/hbase.version
> > HBASE-3749 Master can't exit when open port failed
> >
> > -----邮件原件-----
> > 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel
> Cryans
> > 发送时间: 2011年5月10日 1:17
> > 收件人: user@hbase.apache.org
> > 主题: Re: A question about client
> >
> > TreeMap isn't concurrent and it seems it was used that way? I know you
> > guys are testing a bunch of different things at the same time so which
> > HBase version and which patches were you using when you got that?
> >
> > Thx,
> >
> > J-D
> >
> > On Mon, May 9, 2011 at 5:22 AM, Gaojinchao <ga...@huawei.com>
> wrote:
> >> I used ycsb to put data and threw exception.
> >> Who can give me some suggestion?
> >>
> >> Hbase Code:
> >> // Cut the cache so that we only get the part that could contain
> >> // regions that match our key
> >> SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
> >> tableLocations.headMap(row);
> >>
> >> // if that portion of the map is empty, then we're done. otherwise,
> >> // we need to examine the cached location to verify that it is
> >> // a match by end key as well.
> >> if (!matchingRegions.isEmpty()) {
> >> HRegionLocation possibleRegion =
> >> matchingRegions.get(matchingRegions.lastKey());
> >>
> >> ycsb client log:
> >>
> >> [java] begin StatusThread run
> >> [java] java.util.NoSuchElementException
> >> [java] at java.util.TreeMap.key(TreeMap.java:1206)
> >> [java] at
> java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
> >> [java] at
> org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
> >> [java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
> >> [java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
> >> [java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
> >> [java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown
> Source)
> >> [java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
> >>
> >
>
re: A question about client
Posted by Gaojinchao <ga...@huawei.com>.
Sorry, give other information:
Ycsb don't share the HTables.
One thread has a Htable instance
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2011年5月11日 10:31
收件人: user@hbase.apache.org
主题: Re: A question about client
I think the second explanation is plausible.
From
http://download.oracle.com/javase/1.4.2/docs/api/java/lang/ref/SoftReference.html
:
Soft reference objects, which are cleared at the discretion of the garbage
collector in response to memory demand.
I think we should check whether possibleRegion is null in the code below.
2011/5/10 Jean-Daniel Cryans <jd...@apache.org>
> Are you running a modified YCSB by any chance? Because last time I
> looked at that code it didn't share the HTables between threads and it
> looks like it's doing something like that.
>
> Looking more deeper at the code, the NoSuchElementException is thrown
> because the map is empty. This is what that code looks like:
>
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
>
> So to me it seems that the only way you would get this exception is if
> someone emptied the map between the isEmpty call and lastKey which
> shouldn't happen if HTables aren't shared.
>
> The only other way it seems it could happen, and it's a stretch, is
> that since the regions are kept in a SoftValueSortedMap then the GC
> would have removed the elements you needed exactly between those two
> lines... Is it easy for you to recreate the issue?
>
> Thx a bunch,
>
> J-D
>
> On Mon, May 9, 2011 at 11:34 PM, Gaojinchao <ga...@huawei.com> wrote:
> > Hbase version: 0.90.2 .
> > I merged patches:
> > HBASE-3773 Set ZK max connections much higher in 0.90
> > HBASE-3771 All jsp pages don't clean their HBA
> > HBASE-3783 hbase-0.90.2.jar exists in hbase root and in 'lib/'
> > HBASE-3756 Can't move META or ROOT from shell
> > HBASE-3744 createTable blocks until all regions are out of
> transition
> > HBASE-3712 HTable.close() doesn't shutdown thread pool
> > HBASE-3750 HTablePool.putTable() should call
> tableFactory.releaseHTableInterface() for discarded table
> > HBASE-3722 A lot of data is lost when name node crashed
> > HBASE-3800 If HMaster is started after NN without starting DN in
> Hbase 090.2 then HMaster is not able to start due to AlreadyCreatedException
> for /hbase/hbase.version
> > HBASE-3749 Master can't exit when open port failed
> >
> > -----邮件原件-----
> > 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel
> Cryans
> > 发送时间: 2011年5月10日 1:17
> > 收件人: user@hbase.apache.org
> > 主题: Re: A question about client
> >
> > TreeMap isn't concurrent and it seems it was used that way? I know you
> > guys are testing a bunch of different things at the same time so which
> > HBase version and which patches were you using when you got that?
> >
> > Thx,
> >
> > J-D
> >
> > On Mon, May 9, 2011 at 5:22 AM, Gaojinchao <ga...@huawei.com>
> wrote:
> >> I used ycsb to put data and threw exception.
> >> Who can give me some suggestion?
> >>
> >> Hbase Code:
> >> // Cut the cache so that we only get the part that could contain
> >> // regions that match our key
> >> SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
> >> tableLocations.headMap(row);
> >>
> >> // if that portion of the map is empty, then we're done. otherwise,
> >> // we need to examine the cached location to verify that it is
> >> // a match by end key as well.
> >> if (!matchingRegions.isEmpty()) {
> >> HRegionLocation possibleRegion =
> >> matchingRegions.get(matchingRegions.lastKey());
> >>
> >> ycsb client log:
> >>
> >> [java] begin StatusThread run
> >> [java] java.util.NoSuchElementException
> >> [java] at java.util.TreeMap.key(TreeMap.java:1206)
> >> [java] at
> java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
> >> [java] at
> org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
> >> [java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
> >> [java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
> >> [java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
> >> [java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown
> Source)
> >> [java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
> >>
> >
>
Re: A question about client
Posted by Ted Yu <yu...@gmail.com>.
I think the second explanation is plausible.
From
http://download.oracle.com/javase/1.4.2/docs/api/java/lang/ref/SoftReference.html
:
Soft reference objects, which are cleared at the discretion of the garbage
collector in response to memory demand.
I think we should check whether possibleRegion is null in the code below.
2011/5/10 Jean-Daniel Cryans <jd...@apache.org>
> Are you running a modified YCSB by any chance? Because last time I
> looked at that code it didn't share the HTables between threads and it
> looks like it's doing something like that.
>
> Looking more deeper at the code, the NoSuchElementException is thrown
> because the map is empty. This is what that code looks like:
>
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
>
> So to me it seems that the only way you would get this exception is if
> someone emptied the map between the isEmpty call and lastKey which
> shouldn't happen if HTables aren't shared.
>
> The only other way it seems it could happen, and it's a stretch, is
> that since the regions are kept in a SoftValueSortedMap then the GC
> would have removed the elements you needed exactly between those two
> lines... Is it easy for you to recreate the issue?
>
> Thx a bunch,
>
> J-D
>
> On Mon, May 9, 2011 at 11:34 PM, Gaojinchao <ga...@huawei.com> wrote:
> > Hbase version: 0.90.2 .
> > I merged patches:
> > HBASE-3773 Set ZK max connections much higher in 0.90
> > HBASE-3771 All jsp pages don't clean their HBA
> > HBASE-3783 hbase-0.90.2.jar exists in hbase root and in 'lib/'
> > HBASE-3756 Can't move META or ROOT from shell
> > HBASE-3744 createTable blocks until all regions are out of
> transition
> > HBASE-3712 HTable.close() doesn't shutdown thread pool
> > HBASE-3750 HTablePool.putTable() should call
> tableFactory.releaseHTableInterface() for discarded table
> > HBASE-3722 A lot of data is lost when name node crashed
> > HBASE-3800 If HMaster is started after NN without starting DN in
> Hbase 090.2 then HMaster is not able to start due to AlreadyCreatedException
> for /hbase/hbase.version
> > HBASE-3749 Master can't exit when open port failed
> >
> > -----邮件原件-----
> > 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel
> Cryans
> > 发送时间: 2011年5月10日 1:17
> > 收件人: user@hbase.apache.org
> > 主题: Re: A question about client
> >
> > TreeMap isn't concurrent and it seems it was used that way? I know you
> > guys are testing a bunch of different things at the same time so which
> > HBase version and which patches were you using when you got that?
> >
> > Thx,
> >
> > J-D
> >
> > On Mon, May 9, 2011 at 5:22 AM, Gaojinchao <ga...@huawei.com>
> wrote:
> >> I used ycsb to put data and threw exception.
> >> Who can give me some suggestion?
> >>
> >> Hbase Code:
> >> // Cut the cache so that we only get the part that could contain
> >> // regions that match our key
> >> SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
> >> tableLocations.headMap(row);
> >>
> >> // if that portion of the map is empty, then we're done. otherwise,
> >> // we need to examine the cached location to verify that it is
> >> // a match by end key as well.
> >> if (!matchingRegions.isEmpty()) {
> >> HRegionLocation possibleRegion =
> >> matchingRegions.get(matchingRegions.lastKey());
> >>
> >> ycsb client log:
> >>
> >> [java] begin StatusThread run
> >> [java] java.util.NoSuchElementException
> >> [java] at java.util.TreeMap.key(TreeMap.java:1206)
> >> [java] at
> java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
> >> [java] at
> org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
> >> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
> >> [java] at
> org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
> >> [java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
> >> [java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
> >> [java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
> >> [java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown
> Source)
> >> [java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
> >>
> >
>
Re: A question about client
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Are you running a modified YCSB by any chance? Because last time I
looked at that code it didn't share the HTables between threads and it
looks like it's doing something like that.
Looking more deeper at the code, the NoSuchElementException is thrown
because the map is empty. This is what that code looks like:
if (!matchingRegions.isEmpty()) {
HRegionLocation possibleRegion =
matchingRegions.get(matchingRegions.lastKey());
So to me it seems that the only way you would get this exception is if
someone emptied the map between the isEmpty call and lastKey which
shouldn't happen if HTables aren't shared.
The only other way it seems it could happen, and it's a stretch, is
that since the regions are kept in a SoftValueSortedMap then the GC
would have removed the elements you needed exactly between those two
lines... Is it easy for you to recreate the issue?
Thx a bunch,
J-D
On Mon, May 9, 2011 at 11:34 PM, Gaojinchao <ga...@huawei.com> wrote:
> Hbase version: 0.90.2 .
> I merged patches:
> HBASE-3773 Set ZK max connections much higher in 0.90
> HBASE-3771 All jsp pages don't clean their HBA
> HBASE-3783 hbase-0.90.2.jar exists in hbase root and in 'lib/'
> HBASE-3756 Can't move META or ROOT from shell
> HBASE-3744 createTable blocks until all regions are out of transition
> HBASE-3712 HTable.close() doesn't shutdown thread pool
> HBASE-3750 HTablePool.putTable() should call tableFactory.releaseHTableInterface() for discarded table
> HBASE-3722 A lot of data is lost when name node crashed
> HBASE-3800 If HMaster is started after NN without starting DN in Hbase 090.2 then HMaster is not able to start due to AlreadyCreatedException for /hbase/hbase.version
> HBASE-3749 Master can't exit when open port failed
>
> -----邮件原件-----
> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
> 发送时间: 2011年5月10日 1:17
> 收件人: user@hbase.apache.org
> 主题: Re: A question about client
>
> TreeMap isn't concurrent and it seems it was used that way? I know you
> guys are testing a bunch of different things at the same time so which
> HBase version and which patches were you using when you got that?
>
> Thx,
>
> J-D
>
> On Mon, May 9, 2011 at 5:22 AM, Gaojinchao <ga...@huawei.com> wrote:
>> I used ycsb to put data and threw exception.
>> Who can give me some suggestion?
>>
>> Hbase Code:
>> // Cut the cache so that we only get the part that could contain
>> // regions that match our key
>> SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
>> tableLocations.headMap(row);
>>
>> // if that portion of the map is empty, then we're done. otherwise,
>> // we need to examine the cached location to verify that it is
>> // a match by end key as well.
>> if (!matchingRegions.isEmpty()) {
>> HRegionLocation possibleRegion =
>> matchingRegions.get(matchingRegions.lastKey());
>>
>> ycsb client log:
>>
>> [java] begin StatusThread run
>> [java] java.util.NoSuchElementException
>> [java] at java.util.TreeMap.key(TreeMap.java:1206)
>> [java] at java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
>> [java] at org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
>> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
>> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
>> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
>> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
>> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
>> [java] at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
>> [java] at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
>> [java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
>> [java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
>> [java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
>> [java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
>> [java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown Source)
>> [java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
>>
>
Re: A question about client
Posted by Gaojinchao <ga...@huawei.com>.
Hbase version: 0.90.2 .
I merged patches:
HBASE-3773 Set ZK max connections much higher in 0.90
HBASE-3771 All jsp pages don't clean their HBA
HBASE-3783 hbase-0.90.2.jar exists in hbase root and in 'lib/'
HBASE-3756 Can't move META or ROOT from shell
HBASE-3744 createTable blocks until all regions are out of transition
HBASE-3712 HTable.close() doesn't shutdown thread pool
HBASE-3750 HTablePool.putTable() should call tableFactory.releaseHTableInterface() for discarded table
HBASE-3722 A lot of data is lost when name node crashed
HBASE-3800 If HMaster is started after NN without starting DN in Hbase 090.2 then HMaster is not able to start due to AlreadyCreatedException for /hbase/hbase.version
HBASE-3749 Master can't exit when open port failed
-----邮件原件-----
发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
发送时间: 2011年5月10日 1:17
收件人: user@hbase.apache.org
主题: Re: A question about client
TreeMap isn't concurrent and it seems it was used that way? I know you
guys are testing a bunch of different things at the same time so which
HBase version and which patches were you using when you got that?
Thx,
J-D
On Mon, May 9, 2011 at 5:22 AM, Gaojinchao <ga...@huawei.com> wrote:
> I used ycsb to put data and threw exception.
> Who can give me some suggestion?
>
> Hbase Code:
> // Cut the cache so that we only get the part that could contain
> // regions that match our key
> SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.headMap(row);
>
> // if that portion of the map is empty, then we're done. otherwise,
> // we need to examine the cached location to verify that it is
> // a match by end key as well.
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
>
> ycsb client log:
>
> [java] begin StatusThread run
> [java] java.util.NoSuchElementException
> [java] at java.util.TreeMap.key(TreeMap.java:1206)
> [java] at java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
> [java] at org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
> [java] at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
> [java] at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
> [java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
> [java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
> [java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
> [java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
> [java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown Source)
> [java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
>
Re: A question about client
Posted by Jean-Daniel Cryans <jd...@apache.org>.
TreeMap isn't concurrent and it seems it was used that way? I know you
guys are testing a bunch of different things at the same time so which
HBase version and which patches were you using when you got that?
Thx,
J-D
On Mon, May 9, 2011 at 5:22 AM, Gaojinchao <ga...@huawei.com> wrote:
> I used ycsb to put data and threw exception.
> Who can give me some suggestion?
>
> Hbase Code:
> // Cut the cache so that we only get the part that could contain
> // regions that match our key
> SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
> tableLocations.headMap(row);
>
> // if that portion of the map is empty, then we're done. otherwise,
> // we need to examine the cached location to verify that it is
> // a match by end key as well.
> if (!matchingRegions.isEmpty()) {
> HRegionLocation possibleRegion =
> matchingRegions.get(matchingRegions.lastKey());
>
> ycsb client log:
>
> [java] begin StatusThread run
> [java] java.util.NoSuchElementException
> [java] at java.util.TreeMap.key(TreeMap.java:1206)
> [java] at java.util.TreeMap$NavigableSubMap.lastKey(TreeMap.java:1435)
> [java] at org.apache.hadoop.hbase.util.SoftValueSortedMap.lastKey(SoftValueSortedMap.java:131)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:841)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:664)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
> [java] at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
> [java] at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
> [java] at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
> [java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:665)
> [java] at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
> [java] at com.yahoo.ycsb.db.HBaseClient.insert(Unknown Source)
> [java] at com.yahoo.ycsb.DBWrapper.insert(Unknown Source)
> [java] at com.yahoo.ycsb.workloads.MyWorkload.doInsert(Unknown Source)
> [java] at com.yahoo.ycsb.ClientThread.run(Unknown Source)
>