[jira] [Commented] (HBASE-8804) Received exception while doing rolling-restart

["Jean-Marc Spaggiari (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-8804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693031#comment-13693031 ] 

Jean-Marc Spaggiari commented on HBASE-8804:
--------------------------------------------

I retried the same host and it worked fine. Maybe the region moved to another location. I will retry to redo all the hosts.

Maybe the script did not wait long enought after the restart? Like maybe we should had a 3 or 4 seconds delay? Not sure.

Node6 is my slowest node on the cluster...
                
> Received exception while doing rolling-restart
> ----------------------------------------------
>
>                 Key: HBASE-8804
>                 URL: https://issues.apache.org/jira/browse/HBASE-8804
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Jean-Marc Spaggiari
>
> I got this exception while doing a rolling restart on my cluster:
> {code}
> 13/06/25 09:19:13 INFO region_mover: Wrote list of moved regions to /tmp/node6
> Unloaded node6 region(s)
> node6: stopping regionserver..
> node6: starting regionserver, logging to /home/hbase/hbase-0.94.3/bin/../logs/hbase-hbase-regionserver-node6.out
> Reloading node6 region(s)
> 13/06/25 09:19:20 INFO region_mover: hostname=node6 is not up yet, waiting
> 13/06/25 09:19:20 INFO region_mover: hostname=node6 is not up yet, waiting
> 13/06/25 09:19:21 INFO region_mover: hostname=node6 is not up yet, waiting
> 13/06/25 09:19:21 INFO region_mover: hostname=node6 is not up yet, waiting
> 13/06/25 09:19:23 ERROR io.HbaseObjectWritable: Error in readFields
> java.lang.IllegalArgumentException: Not a host:port pair: de7,60020,1370440536028
> X�
�UIntegrationTestBigLinkedListm�N�|�,1366851785897.6fb3d84e41ffed2f
> 	at org.apache.hadoop.hbase.util.Addressing.parseHostname(Addressing.java:60)
> 	at org.apache.hadoop.hbase.ServerName.<init>(ServerName.java:101)
> 	at org.apache.hadoop.hbase.ServerName.parseVersionedServerName(ServerName.java:283)
> 	at org.apache.hadoop.hbase.ClusterStatus.readFields(ClusterStatus.java:314)
> 	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:690)
> 	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:331)
> 	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:668)
> 	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580)
> NativeException: java.lang.reflect.UndeclaredThrowableException: Call to node3/192.168.23.7:60000 failed on local exception: java.io.IOException: Error in readFields
>    getServers at /home/hbase/hbase-0.94.3/bin/region_mover.rb:193
>   loadRegions at /home/hbase/hbase-0.94.3/bin/region_mover.rb:367
>        (root) at /home/hbase/hbase-0.94.3/bin/region_mover.rb:469
> Reloaded node6 region(s)
> Restoring balancer state to true
> Disabling balancer! (if required)
> {code}
> Ran successfuly on the 6 previous nodes, but failed on this one. This table has 14 regions spreaded over all the RS. The region which failed is "IntegrationTestBigLinkedList,\x08\x06m\xDENxFD|\xD2,1366851785897.6fb3d84e41ffed2f33931e3dac8b114f." and is online on Node2. I'm not really sure what the rolling restart failed. Bad since I will have to re-try it and it takes 2h :(

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira