incorrect host display for RegionServer

[Brian Jeltema <br...@digitalenvoy.net>]

I have a 6-node cluster that I deployed with Ambari. While trying to resolve some code issues, I
manually stopped/restarted some HBase RegionServers. Now the 'Hosts' page shows some
nodes with a yellow bullet and the popup says the RegionServer is down in those nodes. However,
those servers are running and all of the other UI components show those servers as being up.
I've restarted the server and all the agents,  with no change. Any suggestions as to cause/fix?

TIA

Brian

Re: incorrect host display for RegionServer

[Brian Jeltema <br...@digitalenvoy.net>]

Thanks. The pid files were the problem. 

By 'other UI components' I was referring to things like the HBase Master Web UI
and the Alert/Health check window in the HBase service page.

On Jul 10, 2013, at 10:48 AM, Sumit Mohanty wrote:

> The Ambari agents rely on a set of files that store the pid of the running
> process to figure out if the components (e.g. RegionServer) are up. While
> the actual path may vary due to specific deployments, you can look at
> /var/run/hbase to see if the PID in the file is matching the process id of
> the active region server.
> 
> Is it possible for you to shutdown the region servers manually and start
> them through Ambari? If that is not possible then I can dig-up the exact
> command, as used by Ambari, to start the region server. That will ensure
> that the Ambari agent can "see" the region servers.
> 
> Can you provide more details on "other UI components show those servers as
> being up"?
> 
> Thanks
> Sumit
> 
> On 7/10/13 6:05 AM, "Brian Jeltema" <br...@digitalenvoy.net> wrote:
> 
>> I have a 6-node cluster that I deployed with Ambari. While trying to
>> resolve some code issues, I
>> manually stopped/restarted some HBase RegionServers. Now the 'Hosts' page
>> shows some
>> nodes with a yellow bullet and the popup says the RegionServer is down in
>> those nodes. However,
>> those servers are running and all of the other UI components show those
>> servers as being up.
>> I've restarted the server and all the agents,  with no change. Any
>> suggestions as to cause/fix?
>> 
>> TIA
>> 
>> Brian
> 
> 

Re: incorrect host display for RegionServer

[Sumit Mohanty <sm...@hortonworks.com>]

The Ambari agents rely on a set of files that store the pid of the running
process to figure out if the components (e.g. RegionServer) are up. While
the actual path may vary due to specific deployments, you can look at
/var/run/hbase to see if the PID in the file is matching the process id of
the active region server.

Is it possible for you to shutdown the region servers manually and start
them through Ambari? If that is not possible then I can dig-up the exact
command, as used by Ambari, to start the region server. That will ensure
that the Ambari agent can "see" the region servers.
 
Can you provide more details on "other UI components show those servers as
being up"?

Thanks
Sumit

On 7/10/13 6:05 AM, "Brian Jeltema" <br...@digitalenvoy.net> wrote:

>I have a 6-node cluster that I deployed with Ambari. While trying to
>resolve some code issues, I
>manually stopped/restarted some HBase RegionServers. Now the 'Hosts' page
>shows some
>nodes with a yellow bullet and the popup says the RegionServer is down in
>those nodes. However,
>those servers are running and all of the other UI components show those
>servers as being up.
>I've restarted the server and all the agents,  with no change. Any
>suggestions as to cause/fix?
>
>TIA
>
>Brian