You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Pete Tyler <pe...@gmail.com> on 2011/04/22 19:32:04 UTC
Exception after upgrading to 90.1
Seeing this error in the client. I can create new HTable instances fine until
I get to this one unit test, then I can't open HTable instances that I could
open earlier. As far as I can tell the erro starts happening immediately
after my client process has run a map reduce job locally.
Running Hbase 90.1, hadoop-0.20.2-CDH3B4, java 1.6.0_24, OSX 10.6.7
java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51160fa6
closed
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:567)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
--
View this message in context: http://old.nabble.com/Exception-after-upgrading-to-90.1-tp31457860p31457860.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Exception after upgrading to 90.1
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Probably the same ConnectionLossException that others have been
describing on this list? I don't see it in your stack trace (in fact I
can't really see anything), but it sounds like what you describe.
J-D
On Fri, Apr 22, 2011 at 10:32 AM, Pete Tyler <pe...@gmail.com> wrote:
>
> Seeing this error in the client. I can create new HTable instances fine until
> I get to this one unit test, then I can't open HTable instances that I could
> open earlier. As far as I can tell the erro starts happening immediately
> after my client process has run a map reduce job locally.
>
> Running Hbase 90.1, hadoop-0.20.2-CDH3B4, java 1.6.0_24, OSX 10.6.7
>
> java.io.IOException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51160fa6
> closed
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:567)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
>
> --
> View this message in context: http://old.nabble.com/Exception-after-upgrading-to-90.1-tp31457860p31457860.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
Re: Row Key Question
Posted by Jean-Daniel Cryans <jd...@apache.org>.
The splitting is based on when a region reaches a configured size
(default is 256MB). A table starts with 1 region, and splits as needed
when you insert. For a bit more info see:
http://hbase.apache.org/book.html#regions.arch
J-D
On Fri, Apr 22, 2011 at 10:40 AM, Peter Haidinyak <ph...@local.com> wrote:
> I have a question on how HBase decides to save rows based on Row Keys. Say I have a million rows to insert into a new table in a ten node cluster. Each row's key is some random 32 byte value and there are two columns per row, each column contains some random 32 byte value.
> My question is how does HBase know when to 'split' the table between the ten nodes? Or how does HBase 'split' the random keys between the ten nodes?
>
> Thanks
>
> -Pete
>
Re: Splitlog() executed while the namenode was in safemode may cause data-loss
Posted by Stack <st...@duboce.net>.
That sounds like a good idea. If you don't mind, please file an
issue and make a patch.
Thank you,
St.Ack
On Sun, Apr 24, 2011 at 5:53 PM, bijieshan <bi...@huawei.com> wrote:
> Under the current hdfs Version, there's no related method to judge whether the namenode is in safemode.
> Maybe we can handle the SafeModeException at the top layer where called the method of checkFileSystem(), wait for a while and retry the operation.
> Does that reasonable? I hope someone can give some advises.
>
> Thanks!
> Jieshan Bean
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年4月24日 5:55
> 收件人: user@hbase.apache.org
> 抄送: Chenjian
> 主题: Re: Splitlog() executed while the namenode was in safemode may cause data-loss
>
> Sorry, what did you change?
> Thanks,
> St.Ack
>
> On Fri, Apr 22, 2011 at 9:00 PM, bijieshan <bi...@huawei.com> wrote:
>> Hi,
>> I found this problem while the namenode went into safemode due to some unclear reasons.
>> There's one patch about this problem:
>>
>> try {
>> HLogSplitter splitter = HLogSplitter.createLogSplitter(
>> conf, rootdir, logDir, oldLogDir, this.fs);
>> try {
>> splitter.splitLog();
>> } catch (OrphanHLogAfterSplitException e) {
>> LOG.warn("Retrying splitting because of:", e);
>> // An HLogSplitter instance can only be used once. Get new instance.
>> splitter = HLogSplitter.createLogSplitter(conf, rootdir, logDir,
>> oldLogDir, this.fs);
>> splitter.splitLog();
>> }
>> splitTime = splitter.getTime();
>> splitLogSize = splitter.getSize();
>> } catch (IOException e) {
>> checkFileSystem();
>> LOG.error("Failed splitting " + logDir.toString(), e);
>> master.abort("Shutting down HBase cluster: Failed splitting hlog files...", e);
>> } finally {
>> this.splitLogLock.unlock();
>> }
>>
>> And it was really give some useful help to some extent, while the namenode process exited or been killed, but not considered the Namenode safemode exception.
>> I think the root reason is the method of checkFileSystem().
>> It gives out an method to check whether the HDFS works normally(Read and write could be success), and that maybe the original propose of this method. This's how this method implements:
>>
>> DistributedFileSystem dfs = (DistributedFileSystem) fs;
>> try {
>> if (dfs.exists(new Path("/"))) {
>> return;
>> }
>> } catch (IOException e) {
>> exception = RemoteExceptionHandler.checkIOException(e);
>> }
>>
>> I have check the hdfs code, and learned that while the namenode was in safemode ,the dfs.exists(new Path("/")) returned true. Because the file system could provide read-only service. So this method just checks the dfs whether could be read. I
>> think it's not reasonable.
>>
>>
>> Regards,
>> Jieshan Bean
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: Splitlog() executed while the namenode was in safemode may cause
data-loss
Posted by bijieshan <bi...@huawei.com>.
Under the current hdfs Version, there's no related method to judge whether the namenode is in safemode.
Maybe we can handle the SafeModeException at the top layer where called the method of checkFileSystem(), wait for a while and retry the operation.
Does that reasonable? I hope someone can give some advises.
Thanks!
Jieshan Bean
-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月24日 5:55
收件人: user@hbase.apache.org
抄送: Chenjian
主题: Re: Splitlog() executed while the namenode was in safemode may cause data-loss
Sorry, what did you change?
Thanks,
St.Ack
On Fri, Apr 22, 2011 at 9:00 PM, bijieshan <bi...@huawei.com> wrote:
> Hi,
> I found this problem while the namenode went into safemode due to some unclear reasons.
> There's one patch about this problem:
>
> try {
> HLogSplitter splitter = HLogSplitter.createLogSplitter(
> conf, rootdir, logDir, oldLogDir, this.fs);
> try {
> splitter.splitLog();
> } catch (OrphanHLogAfterSplitException e) {
> LOG.warn("Retrying splitting because of:", e);
> // An HLogSplitter instance can only be used once. Get new instance.
> splitter = HLogSplitter.createLogSplitter(conf, rootdir, logDir,
> oldLogDir, this.fs);
> splitter.splitLog();
> }
> splitTime = splitter.getTime();
> splitLogSize = splitter.getSize();
> } catch (IOException e) {
> checkFileSystem();
> LOG.error("Failed splitting " + logDir.toString(), e);
> master.abort("Shutting down HBase cluster: Failed splitting hlog files...", e);
> } finally {
> this.splitLogLock.unlock();
> }
>
> And it was really give some useful help to some extent, while the namenode process exited or been killed, but not considered the Namenode safemode exception.
> I think the root reason is the method of checkFileSystem().
> It gives out an method to check whether the HDFS works normally(Read and write could be success), and that maybe the original propose of this method. This's how this method implements:
>
> DistributedFileSystem dfs = (DistributedFileSystem) fs;
> try {
> if (dfs.exists(new Path("/"))) {
> return;
> }
> } catch (IOException e) {
> exception = RemoteExceptionHandler.checkIOException(e);
> }
>
> I have check the hdfs code, and learned that while the namenode was in safemode ,the dfs.exists(new Path("/")) returned true. Because the file system could provide read-only service. So this method just checks the dfs whether could be read. I
> think it's not reasonable.
>
>
> Regards,
> Jieshan Bean
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Splitlog() executed while the namenode was in safemode may cause data-loss
Posted by Stack <st...@duboce.net>.
Sorry, what did you change?
Thanks,
St.Ack
On Fri, Apr 22, 2011 at 9:00 PM, bijieshan <bi...@huawei.com> wrote:
> Hi,
> I found this problem while the namenode went into safemode due to some unclear reasons.
> There's one patch about this problem:
>
> try {
> HLogSplitter splitter = HLogSplitter.createLogSplitter(
> conf, rootdir, logDir, oldLogDir, this.fs);
> try {
> splitter.splitLog();
> } catch (OrphanHLogAfterSplitException e) {
> LOG.warn("Retrying splitting because of:", e);
> // An HLogSplitter instance can only be used once. Get new instance.
> splitter = HLogSplitter.createLogSplitter(conf, rootdir, logDir,
> oldLogDir, this.fs);
> splitter.splitLog();
> }
> splitTime = splitter.getTime();
> splitLogSize = splitter.getSize();
> } catch (IOException e) {
> checkFileSystem();
> LOG.error("Failed splitting " + logDir.toString(), e);
> master.abort("Shutting down HBase cluster: Failed splitting hlog files...", e);
> } finally {
> this.splitLogLock.unlock();
> }
>
> And it was really give some useful help to some extent, while the namenode process exited or been killed, but not considered the Namenode safemode exception.
> I think the root reason is the method of checkFileSystem().
> It gives out an method to check whether the HDFS works normally(Read and write could be success), and that maybe the original propose of this method. This's how this method implements:
>
> DistributedFileSystem dfs = (DistributedFileSystem) fs;
> try {
> if (dfs.exists(new Path("/"))) {
> return;
> }
> } catch (IOException e) {
> exception = RemoteExceptionHandler.checkIOException(e);
> }
>
> I have check the hdfs code, and learned that while the namenode was in safemode ,the dfs.exists(new Path("/")) returned true. Because the file system could provide read-only service. So this method just checks the dfs whether could be read. I
> think it's not reasonable.
>
>
> Regards,
> Jieshan Bean
>
>
>
>
>
>
>
>
>
>
>
>
>
Splitlog() executed while the namenode was in safemode may cause
data-loss
Posted by bijieshan <bi...@huawei.com>.
Hi,
I found this problem while the namenode went into safemode due to some unclear reasons.
There's one patch about this problem:
try {
HLogSplitter splitter = HLogSplitter.createLogSplitter(
conf, rootdir, logDir, oldLogDir, this.fs);
try {
splitter.splitLog();
} catch (OrphanHLogAfterSplitException e) {
LOG.warn("Retrying splitting because of:", e);
// An HLogSplitter instance can only be used once. Get new instance.
splitter = HLogSplitter.createLogSplitter(conf, rootdir, logDir,
oldLogDir, this.fs);
splitter.splitLog();
}
splitTime = splitter.getTime();
splitLogSize = splitter.getSize();
} catch (IOException e) {
checkFileSystem();
LOG.error("Failed splitting " + logDir.toString(), e);
master.abort("Shutting down HBase cluster: Failed splitting hlog files...", e);
} finally {
this.splitLogLock.unlock();
}
And it was really give some useful help to some extent, while the namenode process exited or been killed, but not considered the Namenode safemode exception.
I think the root reason is the method of checkFileSystem().
It gives out an method to check whether the HDFS works normally(Read and write could be success), and that maybe the original propose of this method. This's how this method implements:
DistributedFileSystem dfs = (DistributedFileSystem) fs;
try {
if (dfs.exists(new Path("/"))) {
return;
}
} catch (IOException e) {
exception = RemoteExceptionHandler.checkIOException(e);
}
I have check the hdfs code, and learned that while the namenode was in safemode ,the dfs.exists(new Path("/")) returned true. Because the file system could provide read-only service. So this method just checks the dfs whether could be read. I
think it's not reasonable.
Regards,
Jieshan Bean
RE: Row Key Question
Posted by Peter Haidinyak <ph...@local.com>.
Thanks for the link, nice doodles :-) He kind of validates my thoughts, sequential key = BAD, but if you must do it use a prefix. I'm hoping that over time the keys will end up having a better distribution and I can still do a scan using a start and end row. I'll see how it distributes on my home system.
-Pete
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Friday, April 22, 2011 1:32 PM
To: user@hbase.apache.org
Subject: Re: Row Key Question
That's almost exactly what mozilla is doing with sorocco (google for
their presentations).
Also you seem to assume things about the region balancer that are, at
least at the moment, untrue:
> Then the assumption is this process would continue until every server in the cluster has on region of data
That's more like the end result rather than the goal.
> Then during retrieval I could the use ten Threads, each would use a Start and End row with their prefix and the query should be distributed evenly out among the server.
Nothing is done to make sure that your regions will be distributed
that way, the last region for each salt key may very well end up on
the same region server. That's why it's better to use more salting.
And have you seen this?
http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/
J-D
On Fri, Apr 22, 2011 at 1:18 PM, Peter Haidinyak <ph...@local.com> wrote:
> Thanks, that's the way I visualized it happening. Then the assumption is this process would continue until every server in the cluster has on region of data (more or less). My underlying question is that I need to store my data with the key starting with the date (YYYY-MM-DD). I know this means I will have hot spots during inserts but make retrieval more efficient by using a scan with start and end rows. I was thinking of adding a prefix number of 00 to 09, for the ten servers. In theory, each server should only end up with one of the prefixes. Then during retrieval I could the use ten Threads, each would use a Start and End row with their prefix and the query should be distributed evenly out among the server. I'm not sure if using ten Thread to insert the data would buy me anything or not. Anyway, I'm going to try this out at home on my own cluster to see how it performs.
>
> Thanks
>
> -Pete
>
> -----Original Message-----
> From: Buttler, David [mailto:buttler1@llnl.gov]
> Sent: Friday, April 22, 2011 12:10 PM
> To: user@hbase.apache.org
> Subject: RE: Row Key Question
>
> Regions split when they are larger than the configuration parameter region size. Your data is small enough to fit on a single region.
>
> Keys are sorted in a region. When a region splits the new regions are about half the size of the original region, and contain half the key space each.
>
> Dave
>
> -----Original Message-----
> From: Peter Haidinyak [mailto:phaidinyak@local.com]
> Sent: Friday, April 22, 2011 10:41 AM
> To: user@hbase.apache.org
> Subject: Row Key Question
>
> I have a question on how HBase decides to save rows based on Row Keys. Say I have a million rows to insert into a new table in a ten node cluster. Each row's key is some random 32 byte value and there are two columns per row, each column contains some random 32 byte value.
> My question is how does HBase know when to 'split' the table between the ten nodes? Or how does HBase 'split' the random keys between the ten nodes?
>
> Thanks
>
> -Pete
>
Re: Row Key Question
Posted by Jean-Daniel Cryans <jd...@apache.org>.
That's almost exactly what mozilla is doing with sorocco (google for
their presentations).
Also you seem to assume things about the region balancer that are, at
least at the moment, untrue:
> Then the assumption is this process would continue until every server in the cluster has on region of data
That's more like the end result rather than the goal.
> Then during retrieval I could the use ten Threads, each would use a Start and End row with their prefix and the query should be distributed evenly out among the server.
Nothing is done to make sure that your regions will be distributed
that way, the last region for each salt key may very well end up on
the same region server. That's why it's better to use more salting.
And have you seen this?
http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/
J-D
On Fri, Apr 22, 2011 at 1:18 PM, Peter Haidinyak <ph...@local.com> wrote:
> Thanks, that's the way I visualized it happening. Then the assumption is this process would continue until every server in the cluster has on region of data (more or less). My underlying question is that I need to store my data with the key starting with the date (YYYY-MM-DD). I know this means I will have hot spots during inserts but make retrieval more efficient by using a scan with start and end rows. I was thinking of adding a prefix number of 00 to 09, for the ten servers. In theory, each server should only end up with one of the prefixes. Then during retrieval I could the use ten Threads, each would use a Start and End row with their prefix and the query should be distributed evenly out among the server. I'm not sure if using ten Thread to insert the data would buy me anything or not. Anyway, I'm going to try this out at home on my own cluster to see how it performs.
>
> Thanks
>
> -Pete
>
> -----Original Message-----
> From: Buttler, David [mailto:buttler1@llnl.gov]
> Sent: Friday, April 22, 2011 12:10 PM
> To: user@hbase.apache.org
> Subject: RE: Row Key Question
>
> Regions split when they are larger than the configuration parameter region size. Your data is small enough to fit on a single region.
>
> Keys are sorted in a region. When a region splits the new regions are about half the size of the original region, and contain half the key space each.
>
> Dave
>
> -----Original Message-----
> From: Peter Haidinyak [mailto:phaidinyak@local.com]
> Sent: Friday, April 22, 2011 10:41 AM
> To: user@hbase.apache.org
> Subject: Row Key Question
>
> I have a question on how HBase decides to save rows based on Row Keys. Say I have a million rows to insert into a new table in a ten node cluster. Each row's key is some random 32 byte value and there are two columns per row, each column contains some random 32 byte value.
> My question is how does HBase know when to 'split' the table between the ten nodes? Or how does HBase 'split' the random keys between the ten nodes?
>
> Thanks
>
> -Pete
>
RE: Row Key Question
Posted by Peter Haidinyak <ph...@local.com>.
Thanks, that's the way I visualized it happening. Then the assumption is this process would continue until every server in the cluster has on region of data (more or less). My underlying question is that I need to store my data with the key starting with the date (YYYY-MM-DD). I know this means I will have hot spots during inserts but make retrieval more efficient by using a scan with start and end rows. I was thinking of adding a prefix number of 00 to 09, for the ten servers. In theory, each server should only end up with one of the prefixes. Then during retrieval I could the use ten Threads, each would use a Start and End row with their prefix and the query should be distributed evenly out among the server. I'm not sure if using ten Thread to insert the data would buy me anything or not. Anyway, I'm going to try this out at home on my own cluster to see how it performs.
Thanks
-Pete
-----Original Message-----
From: Buttler, David [mailto:buttler1@llnl.gov]
Sent: Friday, April 22, 2011 12:10 PM
To: user@hbase.apache.org
Subject: RE: Row Key Question
Regions split when they are larger than the configuration parameter region size. Your data is small enough to fit on a single region.
Keys are sorted in a region. When a region splits the new regions are about half the size of the original region, and contain half the key space each.
Dave
-----Original Message-----
From: Peter Haidinyak [mailto:phaidinyak@local.com]
Sent: Friday, April 22, 2011 10:41 AM
To: user@hbase.apache.org
Subject: Row Key Question
I have a question on how HBase decides to save rows based on Row Keys. Say I have a million rows to insert into a new table in a ten node cluster. Each row's key is some random 32 byte value and there are two columns per row, each column contains some random 32 byte value.
My question is how does HBase know when to 'split' the table between the ten nodes? Or how does HBase 'split' the random keys between the ten nodes?
Thanks
-Pete
RE: Row Key Question
Posted by "Buttler, David" <bu...@llnl.gov>.
Regions split when they are larger than the configuration parameter region size. Your data is small enough to fit on a single region.
Keys are sorted in a region. When a region splits the new regions are about half the size of the original region, and contain half the key space each.
Dave
-----Original Message-----
From: Peter Haidinyak [mailto:phaidinyak@local.com]
Sent: Friday, April 22, 2011 10:41 AM
To: user@hbase.apache.org
Subject: Row Key Question
I have a question on how HBase decides to save rows based on Row Keys. Say I have a million rows to insert into a new table in a ten node cluster. Each row's key is some random 32 byte value and there are two columns per row, each column contains some random 32 byte value.
My question is how does HBase know when to 'split' the table between the ten nodes? Or how does HBase 'split' the random keys between the ten nodes?
Thanks
-Pete
Row Key Question
Posted by Peter Haidinyak <ph...@local.com>.
I have a question on how HBase decides to save rows based on Row Keys. Say I have a million rows to insert into a new table in a ten node cluster. Each row's key is some random 32 byte value and there are two columns per row, each column contains some random 32 byte value.
My question is how does HBase know when to 'split' the table between the ten nodes? Or how does HBase 'split' the random keys between the ten nodes?
Thanks
-Pete
Re: Exception after upgrading to 90.1
Posted by Ted Yu <yu...@gmail.com>.
>> It appears to me that some change in the behavior under the covers of map
>> reduce jobs is now forcing me to recreate, rather than reuse, instances
>> created by HBaseConfiguration.
See HBASE-2925 <https://issues.apache.org/jira/browse/HBASE-2925>
I and Karthick have come up with a solution. See HBASE-3777.
See also my blog:
http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
On Sun, Apr 24, 2011 at 5:59 PM, Pete Tyler <pe...@gmail.com>wrote:
> Apologies, I should have mentioned I do not see a ConnectionLossException
> in
> any log files. This lack of the described exception made me think I had a
> different problem.
>
> The teeny weeny stack trace I get just shows that the 'closed' field in the
> 'HConnectionManager' class is set to false.
>
> java.io.IOException:
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@18862d13
> closed
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:567)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
>
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)'
>
>
> Tracing through the logs and code I found a workaround whereby I should no
> longer reuse the same HBaseConfiguration/Configuration instance throughout
> my test suite. Previously, with HBase 0.20.4 I just created one instance
> of HBaseConfiguration and used this in ctors when creating HTable
> instances.
>
> My first attempt at a fix was to use a new instance of HBaseConfiguration
> every time I created an instance of HTable. This caused OutOfMemoryError
> exceptions to be thrown part way through my test suite.
>
> To get my test suite working again I have now coded the following solution
> which is less than ideal in that normal, predictable logic involves
> catching
> an exception,
>
> HTable htable = null;
>
> try {
> htable = new HTable(conf, tableName);
> }
> catch (IOException) {
> conf = new HBaseConfiguration.create(conf);
> htable = new HTable(conf, tableName);
> }
>
> It appears to me that some change in the behavior under the covers of map
> reduce jobs is now forcing me to recreate, rather than reuse, instances
> created by HBaseConfiguration.
>
> Oh well, at least my code is working in this area, onto the next breaking
> change :)
>
Re: Exception after upgrading to 90.1
Posted by Pete Tyler <pe...@gmail.com>.
Apologies, I should have mentioned I do not see a ConnectionLossException in
any log files. This lack of the described exception made me think I had a
different problem.
The teeny weeny stack trace I get just shows that the 'closed' field in the
'HConnectionManager' class is set to false.
java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@18862d13
closed
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:567)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)'
Tracing through the logs and code I found a workaround whereby I should no
longer reuse the same HBaseConfiguration/Configuration instance throughout
my test suite. Previously, with HBase 0.20.4 I just created one instance
of HBaseConfiguration and used this in ctors when creating HTable
instances.
My first attempt at a fix was to use a new instance of HBaseConfiguration
every time I created an instance of HTable. This caused OutOfMemoryError
exceptions to be thrown part way through my test suite.
To get my test suite working again I have now coded the following solution
which is less than ideal in that normal, predictable logic involves catching
an exception,
HTable htable = null;
try {
htable = new HTable(conf, tableName);
}
catch (IOException) {
conf = new HBaseConfiguration.create(conf);
htable = new HTable(conf, tableName);
}
It appears to me that some change in the behavior under the covers of map
reduce jobs is now forcing me to recreate, rather than reuse, instances
created by HBaseConfiguration.
Oh well, at least my code is working in this area, onto the next breaking
change :)
Re: Exception after upgrading to 90.1
Posted by Jean-Daniel Cryans <jd...@apache.org>.
I'm pretty sure, like I mentioned before, that the issue isn't that a
connection is closed but it's in fact not closed. Threads like those
ones talk about it:
http://search-hadoop.com/m/JFj52oETZn
http://search-hadoop.com/m/Wxcn42PBN9g2
J-D
On Fri, Apr 22, 2011 at 12:16 PM, Pete Tyler <pe...@gmail.com> wrote:
> One job, then a scan. Both from the same JVM. I do want to run multiple jobs from the same client JVM and those tests are failing too.
>
> I'm currently trying to figure out why the job is closing the connection and how I can stop it doing so.
>
> From my iPhone
>
> On Apr 22, 2011, at 12:05 PM, Jean-Daniel Cryans <jd...@apache.org> wrote:
>
>> Which HTable instantiation is giving you the error here?
>>
>> Are you starting multiple jobs from the same jvm?
>>
>> J-D
>>
>> On Fri, Apr 22, 2011 at 11:16 AM, Pete Tyler <pe...@gmail.com> wrote:
>>>
>>> Is it possible my use of map reduce has been rendered invalid / outdated by
>>> the upgrade? It appears to create the expected result but causes follow on
>>> logic in the client to fail as described above.
>>>
>>> CLIENT:
>>>
>>> HBaseConfiguration conf = new HBaseConfiguration()
>>>
>>> Job job = new Job(conf);
>>> job.setJobName("My Native MapReduce");
>>>
>>> Scan scan = new Scan();
>>>
>>> String tableNameIn =
>>> MyHBaseUtils.getDomainTableName(Publisher.class.getName());
>>> String tableNameOut =
>>> MyHBaseUtils.getDomainTableName(Result.class.getName());
>>>
>>> TableMapReduceUtil.initTableMapperJob(tableNameIn, scan,
>>> NativeMapper.class, ImmutableBytesWritable.class,
>>> ImmutableBytesWritable.class, job);
>>> TableMapReduceUtil.initTableReducerJob(tableNameOut,
>>> NativeReducer.class, job);
>>>
>>> job.setOutputFormatClass(TableOutputFormat.class);
>>> job.setNumReduceTasks(1);
>>>
>>> job.waitForCompletion(true);
>>>
>>>
>>> REDUCE:
>>>
>>> public class NativeReducer extends TableReducer<Writable, Writable,
>>> Writable> {
>>>
>>> @Override
>>> public void reduce(Writable key, Iterable<Writable> values,
>>> Context context) throws IOException, InterruptedException {
>>>
>>> String city = Bytes.toString(((ImmutableBytesWritable)
>>> key).get());
>>> int i = 0;
>>> for (Writable value : values) {
>>> i++;
>>> }
>>>
>>> long nextId = this.getNextId(
>>> "MAPREDUCE_RESULT",
>>> MyHBaseUtils.getSequenceTableName()
>>> );
>>> Put p = new Put(Bytes.toBytes(nextId));
>>>
>>> p.add(Constants.DEFAULT_DATA_FAMILY, Bytes.toBytes("job_id"),
>>> Bytes.toBytes(jobParams.getJobId()));
>>> p.add(Constants.DEFAULT_DATA_FAMILY,
>>> Bytes.toBytes("persisted_key"), this.toByteArray(city));
>>> p.add(Constants.DEFAULT_DATA_FAMILY,
>>> Bytes.toBytes("persisted_value"), this.toByteArray(i));
>>>
>>> long version = 0;
>>> p.add(Constants.DEFAULT_CONTROL_FAMILY,
>>> Constants.DEFAULT_VERSION_QUALIFIER,
>>> Bytes.toBytes(version));
>>>
>>> context.write(key, p);
>>>
>>> }
>>> --
>>> View this message in context: http://old.nabble.com/Exception-after-upgrading-to-90.1-tp31457860p31458091.html
>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>
>>>
>
Re: Exception after upgrading to 90.1
Posted by Pete Tyler <pe...@gmail.com>.
One job, then a scan. Both from the same JVM. I do want to run multiple jobs from the same client JVM and those tests are failing too.
I'm currently trying to figure out why the job is closing the connection and how I can stop it doing so.
From my iPhone
On Apr 22, 2011, at 12:05 PM, Jean-Daniel Cryans <jd...@apache.org> wrote:
> Which HTable instantiation is giving you the error here?
>
> Are you starting multiple jobs from the same jvm?
>
> J-D
>
> On Fri, Apr 22, 2011 at 11:16 AM, Pete Tyler <pe...@gmail.com> wrote:
>>
>> Is it possible my use of map reduce has been rendered invalid / outdated by
>> the upgrade? It appears to create the expected result but causes follow on
>> logic in the client to fail as described above.
>>
>> CLIENT:
>>
>> HBaseConfiguration conf = new HBaseConfiguration()
>>
>> Job job = new Job(conf);
>> job.setJobName("My Native MapReduce");
>>
>> Scan scan = new Scan();
>>
>> String tableNameIn =
>> MyHBaseUtils.getDomainTableName(Publisher.class.getName());
>> String tableNameOut =
>> MyHBaseUtils.getDomainTableName(Result.class.getName());
>>
>> TableMapReduceUtil.initTableMapperJob(tableNameIn, scan,
>> NativeMapper.class, ImmutableBytesWritable.class,
>> ImmutableBytesWritable.class, job);
>> TableMapReduceUtil.initTableReducerJob(tableNameOut,
>> NativeReducer.class, job);
>>
>> job.setOutputFormatClass(TableOutputFormat.class);
>> job.setNumReduceTasks(1);
>>
>> job.waitForCompletion(true);
>>
>>
>> REDUCE:
>>
>> public class NativeReducer extends TableReducer<Writable, Writable,
>> Writable> {
>>
>> @Override
>> public void reduce(Writable key, Iterable<Writable> values,
>> Context context) throws IOException, InterruptedException {
>>
>> String city = Bytes.toString(((ImmutableBytesWritable)
>> key).get());
>> int i = 0;
>> for (Writable value : values) {
>> i++;
>> }
>>
>> long nextId = this.getNextId(
>> "MAPREDUCE_RESULT",
>> MyHBaseUtils.getSequenceTableName()
>> );
>> Put p = new Put(Bytes.toBytes(nextId));
>>
>> p.add(Constants.DEFAULT_DATA_FAMILY, Bytes.toBytes("job_id"),
>> Bytes.toBytes(jobParams.getJobId()));
>> p.add(Constants.DEFAULT_DATA_FAMILY,
>> Bytes.toBytes("persisted_key"), this.toByteArray(city));
>> p.add(Constants.DEFAULT_DATA_FAMILY,
>> Bytes.toBytes("persisted_value"), this.toByteArray(i));
>>
>> long version = 0;
>> p.add(Constants.DEFAULT_CONTROL_FAMILY,
>> Constants.DEFAULT_VERSION_QUALIFIER,
>> Bytes.toBytes(version));
>>
>> context.write(key, p);
>>
>> }
>> --
>> View this message in context: http://old.nabble.com/Exception-after-upgrading-to-90.1-tp31457860p31458091.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
Re: Exception after upgrading to 90.1
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Which HTable instantiation is giving you the error here?
Are you starting multiple jobs from the same jvm?
J-D
On Fri, Apr 22, 2011 at 11:16 AM, Pete Tyler <pe...@gmail.com> wrote:
>
> Is it possible my use of map reduce has been rendered invalid / outdated by
> the upgrade? It appears to create the expected result but causes follow on
> logic in the client to fail as described above.
>
> CLIENT:
>
> HBaseConfiguration conf = new HBaseConfiguration()
>
> Job job = new Job(conf);
> job.setJobName("My Native MapReduce");
>
> Scan scan = new Scan();
>
> String tableNameIn =
> MyHBaseUtils.getDomainTableName(Publisher.class.getName());
> String tableNameOut =
> MyHBaseUtils.getDomainTableName(Result.class.getName());
>
> TableMapReduceUtil.initTableMapperJob(tableNameIn, scan,
> NativeMapper.class, ImmutableBytesWritable.class,
> ImmutableBytesWritable.class, job);
> TableMapReduceUtil.initTableReducerJob(tableNameOut,
> NativeReducer.class, job);
>
> job.setOutputFormatClass(TableOutputFormat.class);
> job.setNumReduceTasks(1);
>
> job.waitForCompletion(true);
>
>
> REDUCE:
>
> public class NativeReducer extends TableReducer<Writable, Writable,
> Writable> {
>
> @Override
> public void reduce(Writable key, Iterable<Writable> values,
> Context context) throws IOException, InterruptedException {
>
> String city = Bytes.toString(((ImmutableBytesWritable)
> key).get());
> int i = 0;
> for (Writable value : values) {
> i++;
> }
>
> long nextId = this.getNextId(
> "MAPREDUCE_RESULT",
> MyHBaseUtils.getSequenceTableName()
> );
> Put p = new Put(Bytes.toBytes(nextId));
>
> p.add(Constants.DEFAULT_DATA_FAMILY, Bytes.toBytes("job_id"),
> Bytes.toBytes(jobParams.getJobId()));
> p.add(Constants.DEFAULT_DATA_FAMILY,
> Bytes.toBytes("persisted_key"), this.toByteArray(city));
> p.add(Constants.DEFAULT_DATA_FAMILY,
> Bytes.toBytes("persisted_value"), this.toByteArray(i));
>
> long version = 0;
> p.add(Constants.DEFAULT_CONTROL_FAMILY,
> Constants.DEFAULT_VERSION_QUALIFIER,
> Bytes.toBytes(version));
>
> context.write(key, p);
>
> }
> --
> View this message in context: http://old.nabble.com/Exception-after-upgrading-to-90.1-tp31457860p31458091.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
Re: Exception after upgrading to 90.1
Posted by Pete Tyler <pe...@gmail.com>.
Is it possible my use of map reduce has been rendered invalid / outdated by
the upgrade? It appears to create the expected result but causes follow on
logic in the client to fail as described above.
CLIENT:
HBaseConfiguration conf = new HBaseConfiguration()
Job job = new Job(conf);
job.setJobName("My Native MapReduce");
Scan scan = new Scan();
String tableNameIn =
MyHBaseUtils.getDomainTableName(Publisher.class.getName());
String tableNameOut =
MyHBaseUtils.getDomainTableName(Result.class.getName());
TableMapReduceUtil.initTableMapperJob(tableNameIn, scan,
NativeMapper.class, ImmutableBytesWritable.class,
ImmutableBytesWritable.class, job);
TableMapReduceUtil.initTableReducerJob(tableNameOut,
NativeReducer.class, job);
job.setOutputFormatClass(TableOutputFormat.class);
job.setNumReduceTasks(1);
job.waitForCompletion(true);
REDUCE:
public class NativeReducer extends TableReducer<Writable, Writable,
Writable> {
@Override
public void reduce(Writable key, Iterable<Writable> values,
Context context) throws IOException, InterruptedException {
String city = Bytes.toString(((ImmutableBytesWritable)
key).get());
int i = 0;
for (Writable value : values) {
i++;
}
long nextId = this.getNextId(
"MAPREDUCE_RESULT",
MyHBaseUtils.getSequenceTableName()
);
Put p = new Put(Bytes.toBytes(nextId));
p.add(Constants.DEFAULT_DATA_FAMILY, Bytes.toBytes("job_id"),
Bytes.toBytes(jobParams.getJobId()));
p.add(Constants.DEFAULT_DATA_FAMILY,
Bytes.toBytes("persisted_key"), this.toByteArray(city));
p.add(Constants.DEFAULT_DATA_FAMILY,
Bytes.toBytes("persisted_value"), this.toByteArray(i));
long version = 0;
p.add(Constants.DEFAULT_CONTROL_FAMILY,
Constants.DEFAULT_VERSION_QUALIFIER,
Bytes.toBytes(version));
context.write(key, p);
}
--
View this message in context: http://old.nabble.com/Exception-after-upgrading-to-90.1-tp31457860p31458091.html
Sent from the HBase User mailing list archive at Nabble.com.