You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Serega Sheypak <se...@gmail.com> on 2015/04/05 15:47:35 UTC
DSE 4.6 with OpsCenter 5.1.1, agent can't start, port 9042 is
occupied by DSE
Hi, getting weird problem when agent to connect to OpsCenter
OpsCenter installed on VM with DSE and agent.
It's not for production, I have 3 VMs with DSE and OpsCenter for dev/test
purposes.
The stacktrace from agent log is:
vagrant@dsenode03:~$ sudo cat /var/log/datastax-agent/agent.log
Starting DataStax agent monitor datastax_agent_monitor
INFO [main] 2015-04-05 13:32:31,594 Loading conf files:
/var/lib/datastax-agent/conf/address.yaml
INFO [main] 2015-04-05 13:32:31,642 Java vendor/version: Java HotSpot(TM)
64-Bit Server VM/1.7.0_76
INFO [main] 2015-04-05 13:32:31,642 DataStax Agent version: 5.1.1
INFO [main] 2015-04-05 13:32:31,679 Default config values:
{:cassandra_port 9042, :rollups300_ttl 2419200, :settings_cf "settings",
:restore_req_update_period 60, :my_channel_prefix "/agent", :poll_period
60, :thrift_conn_timeout 10000, :rollups60_ttl 604800, :stomp_port 61620,
:shorttime_interval 10, :longtime_interval 300, :max-seconds-to-sleep 25,
:private-conf-props ["initial_token" "listen_address" "broadcast_address"
"rpc_address"], :thrift_port 9160, :async_retry_timeout 5,
:agent-conf-group "global-cluster-agent-group", :jmx_host "127.0.0.1",
:ec2_metadata_api_host "169.254.169.254", :metrics_enabled 1,
:async_queue_size 5000, :backup_staging_dir nil, :read-buffer-size
10000000, :remote_verify_max 300000, :disk_usage_update_period 60,
:throttle-bytes-per-second 500000, :rollups7200_ttl 31536000,
:remote_backup_retries 3, :ssl_keystore nil, :rollup_snapshot_period 300,
:is_package true, :monitor_command
"/usr/share/datastax-agent/bin/datastax_agent_monitor",
:thrift_socket_timeout 5000, :remote_verify_initial_delay 1000,
:cassandra_log_location "/var/log/cassandra/system.log",
:remote_backup_region "us-west-1", :restore_on_transfer_failure false,
:tmp_dir "/var/lib/datastax-agent/tmp/", :config_md5 nil, :jmx_port 7199,
:write-buffer-size 100000, :jmx_metrics_threadpool_size 4, :use_ssl 0,
:rollups86400_ttl 0, :nodedetails_threadpool_size 3, :api_port 61621,
:kerberos_service nil, :backup_file_queue_max 10000, :jmx_thread_pool_size
5, :production 1, :runs_sudo 1, :max_file_transfer_attempts 30,
:stomp_interface nil, :storage_keyspace "OpsCenter", :hosts ["127.0.0.1"],
:rollup_snapshot_threshold 300, :jmx_retry_timeout 30, :unthrottled-default
10000000000, :remote_backup_retry_delay 5000, :remote_backup_timeout 1000,
:seconds-to-read-kill-channel 0.005, :realtime_interval 5, :pdps_ttl 259200}
INFO [main] 2015-04-05 13:32:31,924 Waiting for the config from OpsCenter
INFO [main] 2015-04-05 13:32:31,925 Attempting to determine Cassandra's
broadcast address through JMX
INFO [Initialization] 2015-04-05 13:32:31,926 New JMX connection (
127.0.0.1:7199)
INFO [main] 2015-04-05 13:32:31,947 Starting Jetty server: {:join? false,
:ssl? false, :host nil, :port 61621}
INFO [Jetty] 2015-04-05 13:32:32,026 Jetty server started
INFO [Initialization] 2015-04-05 13:32:32,054 Using 192.168.56.30 as the
cassandra broadcast address
INFO [Initialization] 2015-04-05 13:32:32,135 cassandra RPC address is nil
INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC address is
192.168.56.30
INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC broadcast address
is 192.168.56.30
ERROR [Initialization] 2015-04-05 13:32:32,342 Can't connect to Cassandra,
retrying
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
tried for query failed (tried: /127.0.0.1:9042
(com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Cannot
connect))
at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:220)
at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1231)
at com.datastax.driver.core.Cluster.init(Cluster.java:158)
at com.datastax.driver.core.Cluster.connect(Cluster.java:246)
at clojurewerkz.cassaforte.client$connect_or_close.doInvoke(client.clj:149)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojurewerkz.cassaforte.client$connect.invoke(client.clj:165)
at opsagent.cassandra$setup_cassandra$fn__2491.invoke(cassandra.clj:269)
at again.core$with_retries_STAR_$fn__2363.invoke(core.clj:98)
at again.core$with_retries_STAR_.invoke(core.clj:97)
at opsagent.cassandra$setup_cassandra.invoke(cassandra.clj:267)
at opsagent.opsagent$setup_cassandra.invoke(opsagent.clj:152)
at opsagent.jmx$determine_ip.invoke(jmx.clj:276)
at opsagent.jmx$setup_jmx$fn__2867.invoke(jmx.clj:293)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
And it tries to repeat in several times.
I did:
vagrant@dsenode03:~$ sudo netstat -alnpt | grep 9042
tcp 0 0 192.168.56.30:*9042* 0.0.0.0:*
LISTEN 5490/java
and see that some stuff already listening to this port
# cut some output
vagrant@dsenode03:~$ sudo lsof -p 5490
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
NAME
java 5490 cassandra cwd DIR 252,0 4096 3670233
/usr/share/dse
java 5490 cassandra rtd DIR 252,0 4096 2 /
# cut some output
vagrant@dsenode03:~$ sudo lsof -p 5490 | grep .log
java 5490 cassandra mem REG 252,0 33554432 3803832
/var/lib/cassandra/commi*tlog*/CommitLog-3-1428238370080*.log*
java 5490 cassandra mem REG 252,0 33554432 3803355
/var/lib/cassandra/commi*tlog*/CommitLog-3-1428238370079*.log*
java 5490 cassandra DEL REG 252,0 3670465
/usr/share/dse/common*/log*4j-1.2.17.jar
java 5490 cassandra DEL REG 252,0 3670467
/usr/share/dse/common/slf4j*-log*4j12-1.7.2.jar
java 5490 cassandra 439w REG 252,0 120225 3802518
/var*/log*/cassandra/system*.log*
Cassandra DSE did occupy this port. What do I do wrong???
I have three nodes and agents work there witout any problems
Re: DSE 4.6 with OpsCenter 5.1.1, agent can't start, port 9042 is
occupied by DSE
Posted by DuyHai Doan <do...@gmail.com>.
I think the problem is with the IP address. Cassandra does listen on
192.168.56.30 and you agent log complains about not being able to connect
to 127.0.0.1. Worth investigating there
On Sun, Apr 5, 2015 at 3:47 PM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, getting weird problem when agent to connect to OpsCenter
> OpsCenter installed on VM with DSE and agent.
> It's not for production, I have 3 VMs with DSE and OpsCenter for dev/test
> purposes.
>
> The stacktrace from agent log is:
>
> vagrant@dsenode03:~$ sudo cat /var/log/datastax-agent/agent.log
>
> Starting DataStax agent monitor datastax_agent_monitor
>
> INFO [main] 2015-04-05 13:32:31,594 Loading conf files:
> /var/lib/datastax-agent/conf/address.yaml
>
> INFO [main] 2015-04-05 13:32:31,642 Java vendor/version: Java HotSpot(TM)
> 64-Bit Server VM/1.7.0_76
>
> INFO [main] 2015-04-05 13:32:31,642 DataStax Agent version: 5.1.1
>
> INFO [main] 2015-04-05 13:32:31,679 Default config values:
> {:cassandra_port 9042, :rollups300_ttl 2419200, :settings_cf "settings",
> :restore_req_update_period 60, :my_channel_prefix "/agent", :poll_period
> 60, :thrift_conn_timeout 10000, :rollups60_ttl 604800, :stomp_port 61620,
> :shorttime_interval 10, :longtime_interval 300, :max-seconds-to-sleep 25,
> :private-conf-props ["initial_token" "listen_address" "broadcast_address"
> "rpc_address"], :thrift_port 9160, :async_retry_timeout 5,
> :agent-conf-group "global-cluster-agent-group", :jmx_host "127.0.0.1",
> :ec2_metadata_api_host "169.254.169.254", :metrics_enabled 1,
> :async_queue_size 5000, :backup_staging_dir nil, :read-buffer-size
> 10000000, :remote_verify_max 300000, :disk_usage_update_period 60,
> :throttle-bytes-per-second 500000, :rollups7200_ttl 31536000,
> :remote_backup_retries 3, :ssl_keystore nil, :rollup_snapshot_period 300,
> :is_package true, :monitor_command
> "/usr/share/datastax-agent/bin/datastax_agent_monitor",
> :thrift_socket_timeout 5000, :remote_verify_initial_delay 1000,
> :cassandra_log_location "/var/log/cassandra/system.log",
> :remote_backup_region "us-west-1", :restore_on_transfer_failure false,
> :tmp_dir "/var/lib/datastax-agent/tmp/", :config_md5 nil, :jmx_port 7199,
> :write-buffer-size 100000, :jmx_metrics_threadpool_size 4, :use_ssl 0,
> :rollups86400_ttl 0, :nodedetails_threadpool_size 3, :api_port 61621,
> :kerberos_service nil, :backup_file_queue_max 10000, :jmx_thread_pool_size
> 5, :production 1, :runs_sudo 1, :max_file_transfer_attempts 30,
> :stomp_interface nil, :storage_keyspace "OpsCenter", :hosts ["127.0.0.1"],
> :rollup_snapshot_threshold 300, :jmx_retry_timeout 30, :unthrottled-default
> 10000000000, :remote_backup_retry_delay 5000, :remote_backup_timeout 1000,
> :seconds-to-read-kill-channel 0.005, :realtime_interval 5, :pdps_ttl 259200}
>
> INFO [main] 2015-04-05 13:32:31,924 Waiting for the config from OpsCenter
>
> INFO [main] 2015-04-05 13:32:31,925 Attempting to determine Cassandra's
> broadcast address through JMX
>
> INFO [Initialization] 2015-04-05 13:32:31,926 New JMX connection (
> 127.0.0.1:7199)
>
> INFO [main] 2015-04-05 13:32:31,947 Starting Jetty server: {:join? false,
> :ssl? false, :host nil, :port 61621}
>
> INFO [Jetty] 2015-04-05 13:32:32,026 Jetty server started
>
> INFO [Initialization] 2015-04-05 13:32:32,054 Using 192.168.56.30 as the
> cassandra broadcast address
>
> INFO [Initialization] 2015-04-05 13:32:32,135 cassandra RPC address is
> nil
>
> INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC address is
> 192.168.56.30
>
> INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC broadcast address
> is 192.168.56.30
>
> ERROR [Initialization] 2015-04-05 13:32:32,342 Can't connect to Cassandra,
> retrying
>
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: /127.0.0.1:9042
> (com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Cannot
> connect))
>
> at
> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:220)
>
> at
> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
>
> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1231)
>
> at com.datastax.driver.core.Cluster.init(Cluster.java:158)
>
> at com.datastax.driver.core.Cluster.connect(Cluster.java:246)
>
> at clojurewerkz.cassaforte.client$connect_or_close.doInvoke(client.clj:149)
>
> at clojure.lang.RestFn.invoke(RestFn.java:410)
>
> at clojurewerkz.cassaforte.client$connect.invoke(client.clj:165)
>
> at opsagent.cassandra$setup_cassandra$fn__2491.invoke(cassandra.clj:269)
>
> at again.core$with_retries_STAR_$fn__2363.invoke(core.clj:98)
>
> at again.core$with_retries_STAR_.invoke(core.clj:97)
>
> at opsagent.cassandra$setup_cassandra.invoke(cassandra.clj:267)
>
> at opsagent.opsagent$setup_cassandra.invoke(opsagent.clj:152)
>
> at opsagent.jmx$determine_ip.invoke(jmx.clj:276)
>
> at opsagent.jmx$setup_jmx$fn__2867.invoke(jmx.clj:293)
>
> at clojure.lang.AFn.run(AFn.java:24)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> And it tries to repeat in several times.
>
>
> I did:
>
> vagrant@dsenode03:~$ sudo netstat -alnpt | grep 9042
>
> tcp 0 0 192.168.56.30:*9042* 0.0.0.0:*
> LISTEN 5490/java
>
>
> and see that some stuff already listening to this port
>
> # cut some output
>
> vagrant@dsenode03:~$ sudo lsof -p 5490
>
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
> NAME
>
> java 5490 cassandra cwd DIR 252,0 4096 3670233
> /usr/share/dse
>
> java 5490 cassandra rtd DIR 252,0 4096 2 /
>
>
> # cut some output
>
> vagrant@dsenode03:~$ sudo lsof -p 5490 | grep .log
>
> java 5490 cassandra mem REG 252,0 33554432 3803832
> /var/lib/cassandra/commi*tlog*/CommitLog-3-1428238370080*.log*
>
> java 5490 cassandra mem REG 252,0 33554432 3803355
> /var/lib/cassandra/commi*tlog*/CommitLog-3-1428238370079*.log*
>
> java 5490 cassandra DEL REG 252,0 3670465
> /usr/share/dse/common*/log*4j-1.2.17.jar
>
> java 5490 cassandra DEL REG 252,0 3670467
> /usr/share/dse/common/slf4j*-log*4j12-1.7.2.jar
>
>
> java 5490 cassandra 439w REG 252,0 120225 3802518
> /var*/log*/cassandra/system*.log*
>
>
> Cassandra DSE did occupy this port. What do I do wrong???
>
> I have three nodes and agents work there witout any problems
>