You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jeff Whiting <je...@qualtrics.com> on 2011/03/28 22:38:25 UTC
Replication and Rebalancer produces a lot of exceptions
My regions weren't very well balanced so I ran the "balancer" in hbase shell and my region servers
starting getting a whole bunch of exceptions like:
2011-03-28 14:33:27,402 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened
response-index,,1300386337049.70eded10f984a3eefab2bcea432121c8.
2011-03-28 14:33:31,847 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
Opening log for replication ds3.hbase.dev.qualtrics.com%3A60020.1301344250501 at 3960
2011-03-28 14:33:31,856 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
currentNbOperations:27 and seenEntries:3 and size: 677
2011-03-28 14:33:31,856 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
Replicating 3
2011-03-28 14:33:31,907 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
Unable to replicate because org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.RuntimeException: org.apache.hadoop.hbase.TableNotFoundException: user-session at
org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
at org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771) at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.replicateLogEntries(Unknown Source)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
2011-03-28 14:33:31,914 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
Replicating 3
2011-03-28 14:33:31,934 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
Unable to replicate because org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.RuntimeException: org.apache.hadoop.hbase.TableNotFoundException: user-session
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy8.replicateLogEntries(Unknown Source)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
Any ideas why I am getting this exception? Is it harmful or will it figure things out eventually?
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: Replication and Rebalancer produces a lot of exceptions
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Sounds good! I'll do that change.
Thx,
J-D
On Tue, Mar 29, 2011 at 4:01 PM, Jeff Whiting <je...@qualtrics.com> wrote:
> I'm not sure...the key to everything was realizing picking up on
> RemoteException in "Unable to replicate because
> org.apache.hadoop.ipc.RemoteException" and realizing that it was on the
> replication cluster.
>
> If it said something like "Unable to replicate. Destination cluster threw an
> exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.lang.RuntimeException:"
>
> Or something that makes it clear it is an exception on the remote or
> destination cluster would be helpful. It is easy to scan over
> org.apache.hadoop.ipc.RemoteException and read it like
> "org.ap...some-kind-of...exception."
>
> ~Jeff
>
> On 3/29/2011 11:35 AM, Jean-Daniel Cryans wrote:
>>
>> Do you think the issue could be better reported to the user?
>>
>> I find the error message obvious, but I've been around that code for a
>> year now :)
>>
>> Thx!
>>
>> J-D
>>
>> On Tue, Mar 29, 2011 at 9:21 AM, Jeff Whiting<je...@qualtrics.com> wrote:
>>>
>>> That was it. I didn't realize that was the exception on the remote
>>> cluster...make sense now.
>>>
>>> ~Jeff
>>>
>>> On 3/28/2011 3:34 PM, Jean-Daniel Cryans wrote:
>>>>
>>>> The slave cluster is saying that the table user-session doesn't
>>>> exist... is it the case?
>>>>
>>>> J-D
>>>>
>>>> On Mon, Mar 28, 2011 at 1:38 PM, Jeff Whiting<je...@qualtrics.com>
>>>> wrote:
>>>>>
>>>>> My regions weren't very well balanced so I ran the "balancer" in hbase
>>>>> shell
>>>>> and my region servers starting getting a whole bunch of exceptions
>>>>> like:
>>>>>
>>>>> 2011-03-28 14:33:27,402 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened
>>>>> response-index,,1300386337049.70eded10f984a3eefab2bcea432121c8.
>>>>> 2011-03-28 14:33:31,847 DEBUG
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>>> Opening
>>>>> log for replication ds3.hbase.dev.qualtrics.com%3A60020.1301344250501
>>>>> at
>>>>> 3960
>>>>> 2011-03-28 14:33:31,856 DEBUG
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>>> currentNbOperations:27 and seenEntries:3 and size: 677
>>>>> 2011-03-28 14:33:31,856 DEBUG
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>>> Replicating 3
>>>>> 2011-03-28 14:33:31,907 WARN
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>>> Unable
>>>>> to replicate because org.apache.hadoop.ipc.RemoteException:
>>>>> java.io.IOException: java.lang.RuntimeException:
>>>>> org.apache.hadoop.hbase.TableNotFoundException: user-session at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
>>>>> at
>>>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
>>>>> at
>>>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
>>>>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>>>>> at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>>>>> at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>>>>> at $Proxy8.replicateLogEntries(Unknown Source)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
>>>>> 2011-03-28 14:33:31,914 DEBUG
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>>> Replicating 3
>>>>> 2011-03-28 14:33:31,934 WARN
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>>> Unable
>>>>> to replicate because org.apache.hadoop.ipc.RemoteException:
>>>>> java.io.IOException: java.lang.RuntimeException:
>>>>> org.apache.hadoop.hbase.TableNotFoundException: user-session
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
>>>>> at
>>>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
>>>>> at
>>>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
>>>>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>>>>> at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>>>> at
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>>>>> at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>>>>> at $Proxy8.replicateLogEntries(Unknown Source)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
>>>>>
>>>>> Any ideas why I am getting this exception? Is it harmful or will it
>>>>> figure
>>>>> things out eventually?
>>>>>
>>>>> Thanks,
>>>>> ~Jeff
>>>>>
>>>>> --
>>>>> Jeff Whiting
>>>>> Qualtrics Senior Software Engineer
>>>>> jeffw@qualtrics.com
>>>>>
>>>>>
>>> --
>>> Jeff Whiting
>>> Qualtrics Senior Software Engineer
>>> jeffw@qualtrics.com
>>>
>>>
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
>
Re: Replication and Rebalancer produces a lot of exceptions
Posted by Jeff Whiting <je...@qualtrics.com>.
I'm not sure...the key to everything was realizing picking up on RemoteException in "Unable to
replicate because org.apache.hadoop.ipc.RemoteException" and realizing that it was on the
replication cluster.
If it said something like "Unable to replicate. Destination cluster threw an exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.RuntimeException:"
Or something that makes it clear it is an exception on the remote or destination cluster would be
helpful. It is easy to scan over org.apache.hadoop.ipc.RemoteException and read it like
"org.ap...some-kind-of...exception."
~Jeff
On 3/29/2011 11:35 AM, Jean-Daniel Cryans wrote:
> Do you think the issue could be better reported to the user?
>
> I find the error message obvious, but I've been around that code for a
> year now :)
>
> Thx!
>
> J-D
>
> On Tue, Mar 29, 2011 at 9:21 AM, Jeff Whiting<je...@qualtrics.com> wrote:
>> That was it. I didn't realize that was the exception on the remote
>> cluster...make sense now.
>>
>> ~Jeff
>>
>> On 3/28/2011 3:34 PM, Jean-Daniel Cryans wrote:
>>> The slave cluster is saying that the table user-session doesn't
>>> exist... is it the case?
>>>
>>> J-D
>>>
>>> On Mon, Mar 28, 2011 at 1:38 PM, Jeff Whiting<je...@qualtrics.com> wrote:
>>>> My regions weren't very well balanced so I ran the "balancer" in hbase
>>>> shell
>>>> and my region servers starting getting a whole bunch of exceptions like:
>>>>
>>>> 2011-03-28 14:33:27,402 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened
>>>> response-index,,1300386337049.70eded10f984a3eefab2bcea432121c8.
>>>> 2011-03-28 14:33:31,847 DEBUG
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>> Opening
>>>> log for replication ds3.hbase.dev.qualtrics.com%3A60020.1301344250501 at
>>>> 3960
>>>> 2011-03-28 14:33:31,856 DEBUG
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>> currentNbOperations:27 and seenEntries:3 and size: 677
>>>> 2011-03-28 14:33:31,856 DEBUG
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>> Replicating 3
>>>> 2011-03-28 14:33:31,907 WARN
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>> Unable
>>>> to replicate because org.apache.hadoop.ipc.RemoteException:
>>>> java.io.IOException: java.lang.RuntimeException:
>>>> org.apache.hadoop.hbase.TableNotFoundException: user-session at
>>>>
>>>> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
>>>> at
>>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
>>>> at
>>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
>>>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>>>
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>>>> at $Proxy8.replicateLogEntries(Unknown Source)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
>>>> 2011-03-28 14:33:31,914 DEBUG
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>> Replicating 3
>>>> 2011-03-28 14:33:31,934 WARN
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>>> Unable
>>>> to replicate because org.apache.hadoop.ipc.RemoteException:
>>>> java.io.IOException: java.lang.RuntimeException:
>>>> org.apache.hadoop.hbase.TableNotFoundException: user-session
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
>>>> at
>>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
>>>> at
>>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
>>>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>>>> at $Proxy8.replicateLogEntries(Unknown Source)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
>>>>
>>>> Any ideas why I am getting this exception? Is it harmful or will it
>>>> figure
>>>> things out eventually?
>>>>
>>>> Thanks,
>>>> ~Jeff
>>>>
>>>> --
>>>> Jeff Whiting
>>>> Qualtrics Senior Software Engineer
>>>> jeffw@qualtrics.com
>>>>
>>>>
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> jeffw@qualtrics.com
>>
>>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: Replication and Rebalancer produces a lot of exceptions
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Do you think the issue could be better reported to the user?
I find the error message obvious, but I've been around that code for a
year now :)
Thx!
J-D
On Tue, Mar 29, 2011 at 9:21 AM, Jeff Whiting <je...@qualtrics.com> wrote:
> That was it. I didn't realize that was the exception on the remote
> cluster...make sense now.
>
> ~Jeff
>
> On 3/28/2011 3:34 PM, Jean-Daniel Cryans wrote:
>>
>> The slave cluster is saying that the table user-session doesn't
>> exist... is it the case?
>>
>> J-D
>>
>> On Mon, Mar 28, 2011 at 1:38 PM, Jeff Whiting<je...@qualtrics.com> wrote:
>>>
>>> My regions weren't very well balanced so I ran the "balancer" in hbase
>>> shell
>>> and my region servers starting getting a whole bunch of exceptions like:
>>>
>>> 2011-03-28 14:33:27,402 DEBUG
>>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened
>>> response-index,,1300386337049.70eded10f984a3eefab2bcea432121c8.
>>> 2011-03-28 14:33:31,847 DEBUG
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>> Opening
>>> log for replication ds3.hbase.dev.qualtrics.com%3A60020.1301344250501 at
>>> 3960
>>> 2011-03-28 14:33:31,856 DEBUG
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>> currentNbOperations:27 and seenEntries:3 and size: 677
>>> 2011-03-28 14:33:31,856 DEBUG
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>> Replicating 3
>>> 2011-03-28 14:33:31,907 WARN
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>> Unable
>>> to replicate because org.apache.hadoop.ipc.RemoteException:
>>> java.io.IOException: java.lang.RuntimeException:
>>> org.apache.hadoop.hbase.TableNotFoundException: user-session at
>>>
>>> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
>>> at
>>>
>>> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
>>> at
>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
>>> at
>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
>>> at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
>>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> at
>>>
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>>
>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>>> at $Proxy8.replicateLogEntries(Unknown Source)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
>>> 2011-03-28 14:33:31,914 DEBUG
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>> Replicating 3
>>> 2011-03-28 14:33:31,934 WARN
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>> Unable
>>> to replicate because org.apache.hadoop.ipc.RemoteException:
>>> java.io.IOException: java.lang.RuntimeException:
>>> org.apache.hadoop.hbase.TableNotFoundException: user-session
>>> at
>>>
>>> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
>>> at
>>>
>>> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
>>> at
>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
>>> at
>>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
>>> at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
>>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> at
>>>
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>>> at $Proxy8.replicateLogEntries(Unknown Source)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
>>> at
>>>
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
>>>
>>> Any ideas why I am getting this exception? Is it harmful or will it
>>> figure
>>> things out eventually?
>>>
>>> Thanks,
>>> ~Jeff
>>>
>>> --
>>> Jeff Whiting
>>> Qualtrics Senior Software Engineer
>>> jeffw@qualtrics.com
>>>
>>>
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
>
Re: Replication and Rebalancer produces a lot of exceptions
Posted by Jeff Whiting <je...@qualtrics.com>.
That was it. I didn't realize that was the exception on the remote cluster...make sense now.
~Jeff
On 3/28/2011 3:34 PM, Jean-Daniel Cryans wrote:
> The slave cluster is saying that the table user-session doesn't
> exist... is it the case?
>
> J-D
>
> On Mon, Mar 28, 2011 at 1:38 PM, Jeff Whiting<je...@qualtrics.com> wrote:
>> My regions weren't very well balanced so I ran the "balancer" in hbase shell
>> and my region servers starting getting a whole bunch of exceptions like:
>>
>> 2011-03-28 14:33:27,402 DEBUG
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened
>> response-index,,1300386337049.70eded10f984a3eefab2bcea432121c8.
>> 2011-03-28 14:33:31,847 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
>> log for replication ds3.hbase.dev.qualtrics.com%3A60020.1301344250501 at
>> 3960
>> 2011-03-28 14:33:31,856 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> currentNbOperations:27 and seenEntries:3 and size: 677
>> 2011-03-28 14:33:31,856 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Replicating 3
>> 2011-03-28 14:33:31,907 WARN
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable
>> to replicate because org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: java.lang.RuntimeException:
>> org.apache.hadoop.hbase.TableNotFoundException: user-session at
>> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
>> at
>> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
>> at
>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
>> at
>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>
>> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> at $Proxy8.replicateLogEntries(Unknown Source)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
>> 2011-03-28 14:33:31,914 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Replicating 3
>> 2011-03-28 14:33:31,934 WARN
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable
>> to replicate because org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: java.lang.RuntimeException:
>> org.apache.hadoop.hbase.TableNotFoundException: user-session
>> at
>> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
>> at
>> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
>> at
>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
>> at
>> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> at $Proxy8.replicateLogEntries(Unknown Source)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
>>
>> Any ideas why I am getting this exception? Is it harmful or will it figure
>> things out eventually?
>>
>> Thanks,
>> ~Jeff
>>
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> jeffw@qualtrics.com
>>
>>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: Replication and Rebalancer produces a lot of exceptions
Posted by Jean-Daniel Cryans <jd...@apache.org>.
The slave cluster is saying that the table user-session doesn't
exist... is it the case?
J-D
On Mon, Mar 28, 2011 at 1:38 PM, Jeff Whiting <je...@qualtrics.com> wrote:
> My regions weren't very well balanced so I ran the "balancer" in hbase shell
> and my region servers starting getting a whole bunch of exceptions like:
>
> 2011-03-28 14:33:27,402 DEBUG
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened
> response-index,,1300386337049.70eded10f984a3eefab2bcea432121c8.
> 2011-03-28 14:33:31,847 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
> log for replication ds3.hbase.dev.qualtrics.com%3A60020.1301344250501 at
> 3960
> 2011-03-28 14:33:31,856 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> currentNbOperations:27 and seenEntries:3 and size: 677
> 2011-03-28 14:33:31,856 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Replicating 3
> 2011-03-28 14:33:31,907 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable
> to replicate because org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: user-session at
> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
> at
> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
> at
> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
> at
> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
> at
> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy8.replicateLogEntries(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
> 2011-03-28 14:33:31,914 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Replicating 3
> 2011-03-28 14:33:31,934 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable
> to replicate because org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: user-session
> at
> org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
> at
> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:129)
> at
> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
> at
> org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:110)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.put(ReplicationSink.java:166)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:142)
> at
> org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:108)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:2710)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy8.replicateLogEntries(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:558)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:340)
>
> Any ideas why I am getting this exception? Is it harmful or will it figure
> things out eventually?
>
> Thanks,
> ~Jeff
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
>