You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Tim Dunphy <bl...@gmail.com> on 2013/12/29 14:21:43 UTC
cassandra monitoring
Hi all,
I'm attempting to configure datastax agent so that opscenter can monitor
cassandra. I am running cassandra 2.0.3 and opscenter-4.0.1-2.noarch
running. Cassandra is running on a centos 5.9 host and the opscenter host
is running on centos 6.5
A ps shows the agent running
[root@beta:~] #ps -ef | grep datastax | grep -v grep
root 2166 1 0 03:31 ? 00:00:00 /bin/bash
/usr/share/datastax-agent/bin/datastax_agent_monitor
106 2187 1 0 03:31 ? 00:01:37
/etc/alternatives/javahome/bin/java -Xmx40M -Xms40M
-Djavax.net.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore
-Djavax.net.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore
-Djavax.net.ssl.keyStorePassword=opscenter
-Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid
-Dlog4j.configuration=/etc/datastax-agent/log4j.properties -jar
datastax-agent-4.0.2-standalone.jar
/var/lib/datastax-agent/conf/address.yaml
And the service itself claims that it is running:
[root@beta:~] #service datastax-agent status
datastax-agent (pid 2187) is running...
On the cassandra node I have ports 61620 and 61621 open on the firewall.
But if I do an lsof and look for those ports I see no activity there.
[root@beta:~] #lsof -i :61620
[root@beta:~] #lsof -i :61621
And a netstat turns up nothing either:
[root@beta:~] #netstat -tapn | egrep "(datastax|ops)"
So I guess it should come as no surprise that the opscenter interface
reports the node as down.
And trying to reinstall the agent remotely by clicking the 'fix' link
errors out:
g is null
If you need to make changes, you can press "Retry" and the installations
will be retried.
And also I got on another attempt:
"Cannot call method 'getRequstStatus' of null.
I'm really wondering why I'm doing wrong here, and how I can work my way
out of this quagmire. It would be beyond awesome to actually get this
working!
I've also attempted to get Cassandra Cluster Admin working. JMX is doing
it's thing on the cassandra node and is running on port 8081. CCA is
running on the same host as the opscenter.
But cca gives me this error once I log in:
Cassandra Cluster Admin <http://cca.jokefire.com/>
Logout <http://cca.jokefire.com/logout.php>
*Fatal error*: Uncaught exception 'TTransportException' with message
'TSocket: timed out reading 4 bytes from beta.jokefire.com:9160' in
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php:268
Stack trace: #0
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87):
TSocket->read(4) #1
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(135):
TTransport->readAll(4) #2
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(102):
TFramedTransport->readFrame() #3
/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87):
TFramedTransport->read(4) #4
/var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(300):
TTransport->readAll(4) #5
/var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(192):
TBinaryProtocol->readI32(NULL) #6
/var/www/Cassandra-Cluster-Admin/include/thrift/packages/cassandra/cassandra.Cassandra.client.php(1017):
TBinaryProtocol->readMessageBegin(NULL, 0, 0) # in
*/var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php* on
line *268*
Any advice I could get on my CCA problem and /or my Opcenter problem would
be great and appreciated.
Thanks
Tim
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
Re: cassandra monitoring
Posted by Nick Bailey <ni...@datastax.com>.
> Install Errored: Failure installing agent on beta.jokefire.com. Error
> output: /var/lib/opscenter/ssl/agentKeyStore.pem: No such file or directory
> Exit code: 1
>
This indicates that there was a problem generating ssl files when OpsCenter
first started up. I would check the log around the first time you started
opscenter for errors. Another option would be to disable ssl communication
between OpsCenter and the agents.
http://www.datastax.com/documentation/opscenter/4.0/webhelp/index.html#opsc/configure/opscConfigSSL_g.html
>
> I was wondering where I could go from here. Also I would like to password
> protect my OpsCenter installation (assuming I can ever get any useful data
> into it). Are there any docs on how I can do that?
>
>
http://www.datastax.com/documentation/opscenter/4.0/webhelp/index.html#opsc/configure/opscConfigureUserAccess_c.html#opscConfigureUserAccess
Re: cassandra monitoring
Posted by "Timothy P. Dunphy" <bl...@jokefire.com>.
Hi Aaron,
You were right. JMX is running on port 7199, it's just the web interface that's on 8081. My mistake. But what I did was to delete my existing cluster and try to build a new cluster within opscenter and try pointing it at my existing cassandra node. Just one node for now, but when we go to production we plan to scale out.
When I tried to install the agent with opscetner, the installation begins but fails with this message a few moments later:
Install Errored: Failure installing agent on beta.jokefire.com. Error output: /var/lib/opscenter/ssl/agentKeyStore.pem: No such file or directory Exit code: 1
I was wondering where I could go from here. Also I would like to password protect my OpsCenter installation (assuming I can ever get any useful data into it). Are there any docs on how I can do that?
Thanks
Tim
----- Original Message -----
From: "Aaron Morton" <aa...@thelastpickle.com>
To: "Cassandra User" <us...@cassandra.apache.org>
Sent: Monday, December 30, 2013 9:19:05 PM
Subject: Re: cassandra monitoring
JMX is doing it's thing on the cassandra node and is running on port 8081
Have you set the JMX port for the cluster in Ops Centre ? The default JMX port has been 7199 for a while.
Off the top of the my head it’s in the same area where you specify the initial nodes in the cluster, maybe behind an “Advanced” button.
The Ops Centre agent talks to the server to find out what JMX port it should use to talk to the local Cassandra install.
Also check the logs in /var/log/datastax
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 30/12/2013, at 2:21 am, Tim Dunphy < bluethundr@gmail.com > wrote:
<blockquote>
Hi all,
I'm attempting to configure datastax agent so that opscenter can monitor cassandra. I am running cassandra 2.0.3 and opscenter-4.0.1-2.noarch running. Cassandra is running on a centos 5.9 host and the opscenter host is running on centos 6.5
A ps shows the agent running
[root@beta:~] #ps -ef | grep datastax | grep -v grep
root 2166 1 0 03:31 ? 00:00:00 /bin/bash /usr/share/datastax-agent/bin/datastax_agent_monitor
106 2187 1 0 03:31 ? 00:01:37 /etc/alternatives/javahome/bin/java -Xmx40M -Xms40M -Djavax.net.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore -Djavax.net.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore -Djavax.net.ssl.keyStorePassword=opscenter -Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid -Dlog4j.configuration=/etc/datastax-agent/log4j.properties -jar datastax-agent-4.0.2-standalone.jar /var/lib/datastax-agent/conf/address.yaml
And the service itself claims that it is running:
[root@beta:~] #service datastax-agent status
datastax-agent (pid 2187) is running...
On the cassandra node I have ports 61620 and 61621 open on the firewall.
But if I do an lsof and look for those ports I see no activity there.
[root@beta:~] #lsof -i :61620
[root@beta:~] #lsof -i :61621
And a netstat turns up nothing either:
[root@beta:~] #netstat -tapn | egrep "(datastax|ops)"
So I guess it should come as no surprise that the opscenter interface reports the node as down.
And trying to reinstall the agent remotely by clicking the 'fix' link errors out:
g is null
If you need to make changes, you can press "Retry" and the installations will be retried.
And also I got on another attempt:
"Cannot call method 'getRequstStatus' of null.
I'm really wondering why I'm doing wrong here, and how I can work my way out of this quagmire. It would be beyond awesome to actually get this working!
I've also attempted to get Cassandra Cluster Admin working. JMX is doing it's thing on the cassandra node and is running on port 8081. CCA is running on the same host as the opscenter.
But cca gives me this error once I log in:
Cassandra Cluster Admin
Logout
Fatal error : Uncaught exception 'TTransportException' with message 'TSocket: timed out reading 4 bytes from beta.jokefire.com:9160 ' in /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php:268 Stack trace: #0 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87): TSocket->read(4) #1 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(135): TTransport->readAll(4) #2 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(102): TFramedTransport->readFrame() #3 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87): TFramedTransport->read(4) #4 /var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(300): TTransport->readAll(4) #5 /var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(192): TBinaryProtocol->readI32(NULL) #6 /var/www/Cassandra-Cluster-Admin/include/thrift/packages/cassandra/cassandra.Cassandra.client.php(1017): TBinaryProtocol->readMessageBegin(NULL, 0, 0) # in /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php on line 268
Any advice I could get on my CCA problem and /or my Opcenter problem would be great and appreciated.
Thanks
Tim
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
</blockquote>
Re: cassandra monitoring
Posted by Aaron Morton <aa...@thelastpickle.com>.
> JMX is doing it's thing on the cassandra node and is running on port 8081
Have you set the JMX port for the cluster in Ops Centre ? The default JMX port has been 7199 for a while.
Off the top of the my head it’s in the same area where you specify the initial nodes in the cluster, maybe behind an “Advanced” button.
The Ops Centre agent talks to the server to find out what JMX port it should use to talk to the local Cassandra install.
Also check the logs in /var/log/datastax
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 30/12/2013, at 2:21 am, Tim Dunphy <bl...@gmail.com> wrote:
> Hi all,
>
> I'm attempting to configure datastax agent so that opscenter can monitor cassandra. I am running cassandra 2.0.3 and opscenter-4.0.1-2.noarch running. Cassandra is running on a centos 5.9 host and the opscenter host is running on centos 6.5
>
> A ps shows the agent running
>
> [root@beta:~] #ps -ef | grep datastax | grep -v grep
> root 2166 1 0 03:31 ? 00:00:00 /bin/bash /usr/share/datastax-agent/bin/datastax_agent_monitor
> 106 2187 1 0 03:31 ? 00:01:37 /etc/alternatives/javahome/bin/java -Xmx40M -Xms40M -Djavax.net.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore -Djavax.net.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore -Djavax.net.ssl.keyStorePassword=opscenter -Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid -Dlog4j.configuration=/etc/datastax-agent/log4j.properties -jar datastax-agent-4.0.2-standalone.jar /var/lib/datastax-agent/conf/address.yaml
>
> And the service itself claims that it is running:
>
> [root@beta:~] #service datastax-agent status
> datastax-agent (pid 2187) is running...
>
> On the cassandra node I have ports 61620 and 61621 open on the firewall.
>
> But if I do an lsof and look for those ports I see no activity there.
>
> [root@beta:~] #lsof -i :61620
> [root@beta:~] #lsof -i :61621
>
> And a netstat turns up nothing either:
> [root@beta:~] #netstat -tapn | egrep "(datastax|ops)"
>
>
> So I guess it should come as no surprise that the opscenter interface reports the node as down.
>
> And trying to reinstall the agent remotely by clicking the 'fix' link errors out:
>
> g is null
>
> If you need to make changes, you can press "Retry" and the installations will be retried.
>
> And also I got on another attempt:
>
> "Cannot call method 'getRequstStatus' of null.
>
> I'm really wondering why I'm doing wrong here, and how I can work my way out of this quagmire. It would be beyond awesome to actually get this working!
>
> I've also attempted to get Cassandra Cluster Admin working. JMX is doing it's thing on the cassandra node and is running on port 8081. CCA is running on the same host as the opscenter.
>
> But cca gives me this error once I log in:
>
> Cassandra Cluster Admin
>
> Logout
>
> Fatal error: Uncaught exception 'TTransportException' with message 'TSocket: timed out reading 4 bytes from beta.jokefire.com:9160' in /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php:268 Stack trace: #0 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87): TSocket->read(4) #1 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(135): TTransport->readAll(4) #2 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TFramedTransport.php(102): TFramedTransport->readFrame() #3 /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TTransport.php(87): TFramedTransport->read(4) #4 /var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(300): TTransport->readAll(4) #5 /var/www/Cassandra-Cluster-Admin/include/thrift/protocol/TBinaryProtocol.php(192): TBinaryProtocol->readI32(NULL) #6 /var/www/Cassandra-Cluster-Admin/include/thrift/packages/cassandra/cassandra.Cassandra.client.php(1017): TBinaryProtocol->readMessageBegin(NULL, 0, 0) # in /var/www/Cassandra-Cluster-Admin/include/thrift/transport/TSocket.php on line 268
>
> Any advice I could get on my CCA problem and /or my Opcenter problem would be great and appreciated.
>
> Thanks
> Tim
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>