You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "mohit.kaushik" <mo...@orkash.com> on 2016/01/08 11:12:19 UTC
Accumulo monitor not coming up
The Accumulo cluster is working fine(all other processes running) but
yesterday I restarted one node on which hadoop datanode, zookeeper node,
tserver, monitor were running. I started all the processes again but
Monitor did not start. I stoped the cluster and restarted but monitor
still did not appear.
I have JMX configured for monitor process. Visual vm showing everything
like monitor is running.. When I start it again from the master server
with command "*start-server.sh ip monitor*" it says monitor is already
running with pid. I can check the process is running with " *ps -ef |
grep pid*" But as port 50095 is not running simply I can not access the
in browser too. I have checked for the firewalls are stopped.
And not only on specific server if I start monitor on other servers the
behaviour is exactly same either JMX is configured or not. Logs have no
error. I also have checked there is no memory issue.
One more strange behaviour noticed, If I stop cluster it does not stop
cleanly and I have to press ctrl+c and when I start the cluster after
start the login prompt hangs forever and I have open another window and
again ssh to master again to work on it.
Please suggest, I am using Apache Accumulo 1.7.0.
Thanks
Mohit Kaushik
Re: Accumulo monitor not coming up
Posted by Josh Elser <jo...@gmail.com>.
I apologize in advance for that ugly command :)
It really should be as simple as `./stop-server.sh <host> <process>`.
There's a new start-daemon.sh script in 1.8. Maybe we can make a better
invocation around stopping a process in 1.8 too.
mohit.kaushik wrote:
> I was using /*"*//*./stop-server.sh $hostname monitor"*/ to stop monitor
> process
> But I should have used
> /*"./stop-server.sh $hostname $ACCUMULO_HOME/.*/accumulo-start.*.jar
> monitor TERM*/"
> which I found in the stop-all.sh script.
Re: Accumulo monitor not coming up
Posted by "mohit.kaushik" <mo...@orkash.com>.
I think, I have ran mutiple monitor processes, perhaps in some version
of 1.5. I was forgotten about the hotstandby nature of Monitor process.
And yes I was using a wrong command :(. Thanks for pointing.
I was using /*"*//*./stop-server.sh $hostname monitor"*/ to stop monitor
process
But I should have used
/*"./stop-server.sh $hostname $ACCUMULO_HOME/.*/accumulo-start.*.jar
monitor TERM*/"
which I found in the stop-all.sh script.
I suppose there were more than two processes running when I tried to
stop with the command It didn't stop so I killed that on server orkash5
and started on server orkash1. But it was already running on some other
server which I was restricting it.
When I killed the process today it moved to another which I confirmed
with `get /accumuo/<your_instance_id>/monitor/http_addr` thanks.
This seems resolved :)
-Mohit Kaushik
On 01/13/2016 09:30 PM, Josh Elser wrote:
> mohit.kaushik wrote:
>> But found it running on one of the server after a day which is not the
>> configured server for monitor process
>
> That's... strange. We don't automatically start any processes, so I'd
> recommend you investigate what commands you're running :)
>
> The Monitor, like the Master, is designed to operated only when one
> instance is running. If you have multiple Monitors running, you might,
> for example, see the "Recent Logs" split across multiple instances. As
> such, we use ZooKeeper as a barrier to prevent multiple Monitor
> servers from running. This let's us have some "hot-standby" instances
> for the cluster which enable you to avoid any downtime (e.g. if an
> operations team is relying on the monitor to be running)
>
> As such, it's possible that you'll see a monitor process running, but
> the web server inside the process is not. You can check ZooKeeper
> (zkCli.sh) for the active Monitor:
>
> `get /accumuo/<your_instance_id>/monitor/http_addr`
>
> > Please also tell when are you planning to release Accumulo-1.7.1 as
> I am also facing some issues which i found fixed there. when I
> checked, It was expected to release on 15-Dec-2015 but Perhaps delayed
> for the last left bug fix, I suppose.
>
> 1.7.1 is on it's way out. We don't have firm dates that software will
> be released, so the best I can give is "soon".
>
> After we get the open issues on JIRA cleaned up, we'll have to run it
> through some testing. Hopefully we can get it out the door while it's
> still January.
>
>
Re: Accumulo monitor not coming up
Posted by Josh Elser <jo...@gmail.com>.
mohit.kaushik wrote:
> But found it running on one of the server after a day which is not the
> configured server for monitor process
That's... strange. We don't automatically start any processes, so I'd
recommend you investigate what commands you're running :)
The Monitor, like the Master, is designed to operated only when one
instance is running. If you have multiple Monitors running, you might,
for example, see the "Recent Logs" split across multiple instances. As
such, we use ZooKeeper as a barrier to prevent multiple Monitor servers
from running. This let's us have some "hot-standby" instances for the
cluster which enable you to avoid any downtime (e.g. if an operations
team is relying on the monitor to be running)
As such, it's possible that you'll see a monitor process running, but
the web server inside the process is not. You can check ZooKeeper
(zkCli.sh) for the active Monitor:
`get /accumuo/<your_instance_id>/monitor/http_addr`
> Please also tell when are you planning to release Accumulo-1.7.1 as I
am also facing some issues which i found fixed there. when I checked, It
was expected to release on 15-Dec-2015 but Perhaps delayed for the last
left bug fix, I suppose.
1.7.1 is on it's way out. We don't have firm dates that software will be
released, so the best I can give is "soon".
After we get the open issues on JIRA cleaned up, we'll have to run it
through some testing. Hopefully we can get it out the door while it's
still January.
Re: Accumulo monitor not coming up
Posted by "mohit.kaushik" <mo...@orkash.com>.
I also checked with the command "*netstat -nape | grep 50095*". I am
sure no other process is using the port, actually its behaviour is
strange sometimes. I tried to start monitor process on different system
as I was unable to start it on the desired server and found the process
did not start on any of them at that time. But found it running on one
of the server after a day which is not the configured server for monitor
process. It has happened for two servers on different days and
interestingly if I stop and start process, It never starts again. What
could be restricting it, goes over to my head. And still I am unable to
start it on desired server :( , I need help.
Please also tell when are you planning to release*Accumulo-1.7.1* as I
am also facing some issues which i found fixed there. when I checked, It
was expected to release on 15-Dec-2015 but Perhaps delayed for the last
left bug fix, I suppose.
Thanks for the reply :)
-Mohit Kaushik
On 01/10/2016 02:59 AM, Josh Elser wrote:
>
> Assuming this is the case, it may be interesting to you to know that
> we changed the default port from 50095 to 9995 in the upcoming
> Accumulo 1.8.
>
> On Jan 8, 2016 6:45 AM, "Eric Newton" <eric.newton@gmail.com
> <ma...@gmail.com>> wrote:
>
> It's possible some other process grabbed port 50095, and that is
> preventing the monitor from starting. In future releases, the
> default port for the monitor will be in a lower range to avoid
> this very problem.
>
> Try running "netstat -nape | grep 50095".
>
> This should show you information about network sockets and
> connections to port 50095. The very end of the line will provide
> process information (pid).
>
> If you provide your master ".debug" logs, it might help to explain
> the shutdown issues you are experiencing.
>
> -Eric
>
>
> On Fri, Jan 8, 2016 at 5:12 AM, mohit.kaushik
> <mohit.kaushik@orkash.com <ma...@orkash.com>> wrote:
>
>
> The Accumulo cluster is working fine(all other processes
> running) but yesterday I restarted one node on which hadoop
> datanode, zookeeper node, tserver, monitor were running. I
> started all the processes again but Monitor did not start. I
> stoped the cluster and restarted but monitor still did not
> appear.
>
> I have JMX configured for monitor process. Visual vm showing
> everything like monitor is running.. When I start it again
> from the master server with command "*start-server.sh ip
> monitor*" it says monitor is already running with pid. I can
> check the process is running with " *ps -ef | grep pid*" But
> as port 50095 is not running simply I can not access the in
> browser too. I have checked for the firewalls are stopped.
>
> And not only on specific server if I start monitor on other
> servers the behaviour is exactly same either JMX is configured
> or not. Logs have no error. I also have checked there is no
> memory issue.
> One more strange behaviour noticed, If I stop cluster it does
> not stop cleanly and I have to press ctrl+c and when I start
> the cluster after start the login prompt hangs forever and I
> have open another window and again ssh to master again to work
> on it.
>
> Please suggest, I am using Apache Accumulo 1.7.0.
>
> Thanks
> Mohit Kaushik
>
>
Re: Accumulo monitor not coming up
Posted by Josh Elser <jo...@gmail.com>.
Assuming this is the case, it may be interesting to you to know that we
changed the default port from 50095 to 9995 in the upcoming Accumulo 1.8.
On Jan 8, 2016 6:45 AM, "Eric Newton" <er...@gmail.com> wrote:
> It's possible some other process grabbed port 50095, and that is
> preventing the monitor from starting. In future releases, the default port
> for the monitor will be in a lower range to avoid this very problem.
>
> Try running "netstat -nape | grep 50095".
>
> This should show you information about network sockets and connections to
> port 50095. The very end of the line will provide process information (pid).
>
> If you provide your master ".debug" logs, it might help to explain the
> shutdown issues you are experiencing.
>
> -Eric
>
>
> On Fri, Jan 8, 2016 at 5:12 AM, mohit.kaushik <mo...@orkash.com>
> wrote:
>
>>
>> The Accumulo cluster is working fine(all other processes running) but
>> yesterday I restarted one node on which hadoop datanode, zookeeper node,
>> tserver, monitor were running. I started all the processes again but
>> Monitor did not start. I stoped the cluster and restarted but monitor still
>> did not appear.
>>
>> I have JMX configured for monitor process. Visual vm showing everything
>> like monitor is running.. When I start it again from the master server with
>> command "*start-server.sh ip monitor*" it says monitor is already
>> running with pid. I can check the process is running with " *ps -ef |
>> grep pid*" But as port 50095 is not running simply I can not access the
>> in browser too. I have checked for the firewalls are stopped.
>>
>> And not only on specific server if I start monitor on other servers the
>> behaviour is exactly same either JMX is configured or not. Logs have no
>> error. I also have checked there is no memory issue.
>> One more strange behaviour noticed, If I stop cluster it does not stop
>> cleanly and I have to press ctrl+c and when I start the cluster after start
>> the login prompt hangs forever and I have open another window and again ssh
>> to master again to work on it.
>>
>> Please suggest, I am using Apache Accumulo 1.7.0.
>>
>> Thanks
>> Mohit Kaushik
>>
>>
>
Re: Accumulo monitor not coming up
Posted by Eric Newton <er...@gmail.com>.
It's possible some other process grabbed port 50095, and that is preventing
the monitor from starting. In future releases, the default port for the
monitor will be in a lower range to avoid this very problem.
Try running "netstat -nape | grep 50095".
This should show you information about network sockets and connections to
port 50095. The very end of the line will provide process information (pid).
If you provide your master ".debug" logs, it might help to explain the
shutdown issues you are experiencing.
-Eric
On Fri, Jan 8, 2016 at 5:12 AM, mohit.kaushik <mo...@orkash.com>
wrote:
>
> The Accumulo cluster is working fine(all other processes running) but
> yesterday I restarted one node on which hadoop datanode, zookeeper node,
> tserver, monitor were running. I started all the processes again but
> Monitor did not start. I stoped the cluster and restarted but monitor still
> did not appear.
>
> I have JMX configured for monitor process. Visual vm showing everything
> like monitor is running.. When I start it again from the master server with
> command "*start-server.sh ip monitor*" it says monitor is already running
> with pid. I can check the process is running with " *ps -ef | grep pid*"
> But as port 50095 is not running simply I can not access the in browser
> too. I have checked for the firewalls are stopped.
>
> And not only on specific server if I start monitor on other servers the
> behaviour is exactly same either JMX is configured or not. Logs have no
> error. I also have checked there is no memory issue.
> One more strange behaviour noticed, If I stop cluster it does not stop
> cleanly and I have to press ctrl+c and when I start the cluster after start
> the login prompt hangs forever and I have open another window and again ssh
> to master again to work on it.
>
> Please suggest, I am using Apache Accumulo 1.7.0.
>
> Thanks
> Mohit Kaushik
>
>