You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by xi...@sky-data.cn on 2018/05/05 08:27:47 UTC
metrics can not work as expected
Hi!
I tested installation on my vbox, it works well, then i move it to a server which run many services.
When install, it failed at check service status.
I checked ambari-metrics-collector.log and found below error:
2018-05-05 08:16:03,536 INFO org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix
2018-05-05 08:16:03,559 WARN org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
....
2018-05-05 08:18:50,342 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-1.novalocal/192.168.80.34:2188, initiating session
2018-05-05 08:18:50,343 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-05-05 08:18:50,811 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server dx-2.novalocal/192.168.80.27:2188. Will not attempt to authenticate using SASL (unknown error)
2018-05-05 08:18:50,812 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-2.novalocal/192.168.80.27:2188, initiating session
2018-05-05 08:18:50,813 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
....
2018-05-05 08:18:50,947 INFO org.apache.helix.manager.zk.ZkClient: Closing zkclient: State:CONNECTING sessionid:0x0 local:null remoteserver:null lastZxid:0 xid:1 sent:35 recv:0 queuedpkts:0 pendingresp:0 queuedevents:0
2018-05-05 08:18:50,947 INFO org.I0Itec.zkclient.ZkEventThread: Terminate ZkClient event thread.
2018-05-05 08:18:51,352 INFO org.apache.zookeeper.ZooKeeper: Session: 0x0 closed
2018-05-05 08:18:51,352 INFO org.apache.helix.manager.zk.ZkClient: Closed zkclient
2018-05-05 08:18:51,352 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 30000
2018-05-05 08:18:51,354 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore failed in state INITED; cause: org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.initializeSubsystem(HBaseTimelineMetricStore.java:118)
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.serviceInit(HBaseTimelineMetricStore.java:96)
.....
The 2188 port is my setting about zookeeper, and from web UI, its status is OK.
Also i check zookeeper process, it runs as usual.
By the way, metrics needs listen 6188 port, but i use `netstat` to check, related processes do not start.
I do not know how to fix it, could someone help me?
Thanks
Re: metrics can not work as expected
Posted by xi...@sky-data.cn.
I reinstalled and tested, found that zookeeper failed indeed:
2018-05-09 14:55:09,187 - File['/var/lib/ambari-agent/tmp/zkSmoke.out'] {'action': ['delete']}
2018-05-09 14:55:09,188 - File['/var/lib/ambari-agent/tmp/zkSmoke.sh'] {'content': StaticFile('zkSmoke.sh'), 'mode': 0755}
2018-05-09 14:55:09,189 - Writing File['/var/lib/ambari-agent/tmp/zkSmoke.sh'] because it doesn't exist
2018-05-09 14:55:09,189 - Changing permission for /var/lib/ambari-agent/tmp/zkSmoke.sh from 644 to 755
2018-05-09 14:55:09,189 - Execute['/var/lib/ambari-agent/tmp/zkSmoke.sh /usr/hdp/current/zookeeper-client/bin/zkCli.sh ambari-qa /etc/zookeeper/conf 2181 False kinit no_keytab no_principal /var/lib/ambari-agent/tmp/zkSmoke.out'] {'logoutput': True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'try_sleep': 5}
zk_node1=dx-app.novalocal
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:63)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:446)
at org.apache.zookeeper.ZooKeeperMain.connectToZK(ZooKeeperMain.java:279)
at org.apache.zookeeper.ZooKeeperMain.<init>(ZooKeeperMain.java:293)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:286)
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:63)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:446)
at org.apache.zookeeper.ZooKeeperMain.connectToZK(ZooKeeperMain.java:279)
at org.apache.zookeeper.ZooKeeperMain.<init>(ZooKeeperMain.java:293)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:286)
Running test on host dx-app.novalocal
Connecting to dx-app.novalocal:2181
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:63)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:446)
at org.apache.zookeeper.ZooKeeperMain.connectToZK(ZooKeeperMain.java:279)
at org.apache.zookeeper.ZooKeeperMain.<init>(ZooKeeperMain.java:293)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:286)
I know little about zookeeper, what is wrong with zookeeper?
Maybe start right but can not work as expected.
From: "Siddharth Wagle" <sw...@hortonworks.com>
To: "user" <us...@ambari.apache.org>
Sent: Wednesday, May 9, 2018 1:26:39 PM
Subject: Re: metrics can not work as expected
Yes the default that Ambari sets in most cases should just work, not sure about the move part, usually configs are cluster specific so cannot know for sure unless diving deeper.
The wiki has some useful info that you could take a look at:
https://cwiki.apache.org/confluence/display/AMBARI/Troubleshooting+Guide
https://cwiki.apache.org/confluence/display/AMBARI/Known+Issues
https://cwiki.apache.org/confluence/display/AMBARI/Configuration
- Sid
From: David Quiroga <qu...@gmail.com>
Sent: Tuesday, May 8, 2018 9:46 PM
To: user@ambari.apache.org
Subject: Re: metrics can not work as expected
In most cases Ambari Metrics runs separate HBase and Zookeeper instances.
The default ports of Ambari Metrics Zookeeper are typically in the 60,000 range.
Would expect the Ambari defaults to do the trick, which I suspect relates to Sid's question.
On Sun, May 6, 2018 at 5:07 AM, < xiang.dai@sky-data.cn > wrote:
I just download them and make my own repo including them.
Then, i install them with ambari UI.
From: "Siddharth Wagle" < swagle@hortonworks.com >
To: "user" < user@ambari.apache.org >
Sent: Saturday, May 5, 2018 11:07:12 PM
Subject: Re: metrics can not work as expected
Collector is not able to reach Zookeeper. Are you not installing AMS using Ambari?
- Sid
From: xiang.dai@sky-data.cn < xiang.dai@sky-data.cn >
Sent: Saturday, May 5, 2018 1:27 AM
To: user
Subject: metrics can not work as expected
Hi!
I tested installation on my vbox, it works well, then i move it to a server which run many services.
When install, it failed at check service status.
I checked ambari-metrics-collector.log and found below error:
2018-05-05 08:16:03,536 INFO org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix
2018-05-05 08:16:03,559 WARN org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
....
2018-05-05 08:18:50,342 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-1.novalocal/ 192.168.80.34:2188 , initiating session
2018-05-05 08:18:50,343 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-05-05 08:18:50,811 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server dx-2.novalocal/ 192.168.80.27:2188 . Will not attempt to authenticate using SASL (unknown error)
2018-05-05 08:18:50,812 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-2.novalocal/ 192.168.80.27:2188 , initiating session
2018-05-05 08:18:50,813 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
....
2018-05-05 08:18:50,947 INFO org.apache.helix.manager.zk.ZkClient: Closing zkclient: State:CONNECTING sessionid:0x0 local:null remoteserver:null lastZxid:0 xid:1 sent:35 recv:0 queuedpkts:0 pendingresp:0 queuedevents:0
2018-05-05 08:18:50,947 INFO org.I0Itec.zkclient.ZkEventThread: Terminate ZkClient event thread.
2018-05-05 08:18:51,352 INFO org.apache.zookeeper.ZooKeeper: Session: 0x0 closed
2018-05-05 08:18:51,352 INFO org.apache.helix.manager.zk.ZkClient: Closed zkclient
2018-05-05 08:18:51,352 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 30000
2018-05-05 08:18:51,354 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore failed in state INITED; cause: org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.initializeSubsystem(HBaseTimelineMetricStore.java:118)
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.serviceInit(HBaseTimelineMetricStore.java:96)
.....
The 2188 port is my setting about zookeeper, and from web UI, its status is OK.
Also i check zookeeper process, it runs as usual.
By the way, metrics needs listen 6188 port, but i use `netstat` to check, related processes do not start.
I do not know how to fix it, could someone help me?
Thanks
--
戴翔
南京天数信息科技有限公司
电话: +86 1 3382776490
公司官网: www.sky-data.cn
免费使用天数润科智能计算平台 SkyDiscovery
Re: metrics can not work as expected
Posted by Siddharth Wagle <sw...@hortonworks.com>.
Yes the default that Ambari sets in most cases should just work, not sure about the move part, usually configs are cluster specific so cannot know for sure unless diving deeper.
The wiki has some useful info that you could take a look at:
https://cwiki.apache.org/confluence/display/AMBARI/Troubleshooting+Guide
https://cwiki.apache.org/confluence/display/AMBARI/Known+Issues
https://cwiki.apache.org/confluence/display/AMBARI/Configuration
- Sid
________________________________
From: David Quiroga <qu...@gmail.com>
Sent: Tuesday, May 8, 2018 9:46 PM
To: user@ambari.apache.org
Subject: Re: metrics can not work as expected
In most cases Ambari Metrics runs separate HBase and Zookeeper instances.
The default ports of Ambari Metrics Zookeeper are typically in the 60,000 range.
Would expect the Ambari defaults to do the trick, which I suspect relates to Sid's question.
On Sun, May 6, 2018 at 5:07 AM, <xi...@sky-data.cn>> wrote:
I just download them and make my own repo including them.
Then, i install them with ambari UI.
________________________________
From: "Siddharth Wagle" <sw...@hortonworks.com>>
To: "user" <us...@ambari.apache.org>>
Sent: Saturday, May 5, 2018 11:07:12 PM
Subject: Re: metrics can not work as expected
Collector is not able to reach Zookeeper. Are you not installing AMS using Ambari?
- Sid
________________________________
From: xiang.dai@sky-data.cn<ma...@sky-data.cn> <xi...@sky-data.cn>>
Sent: Saturday, May 5, 2018 1:27 AM
To: user
Subject: metrics can not work as expected
Hi!
I tested installation on my vbox, it works well, then i move it to a server which run many services.
When install, it failed at check service status.
I checked ambari-metrics-collector.log and found below error:
2018-05-05 08:16:03,536 INFO org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix
2018-05-05 08:16:03,559 WARN org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
....
2018-05-05 08:18:50,342 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-1.novalocal/192.168.80.34:2188<http://192.168.80.34:2188>, initiating session
2018-05-05 08:18:50,343 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-05-05 08:18:50,811 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server dx-2.novalocal/192.168.80.27:2188<http://192.168.80.27:2188>. Will not attempt to authenticate using SASL (unknown error)
2018-05-05 08:18:50,812 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-2.novalocal/192.168.80.27:2188<http://192.168.80.27:2188>, initiating session
2018-05-05 08:18:50,813 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
....
2018-05-05 08:18:50,947 INFO org.apache.helix.manager.zk.ZkClient: Closing zkclient: State:CONNECTING sessionid:0x0 local:null remoteserver:null lastZxid:0 xid:1 sent:35 recv:0 queuedpkts:0 pendingresp:0 queuedevents:0
2018-05-05 08:18:50,947 INFO org.I0Itec.zkclient.ZkEventThread: Terminate ZkClient event thread.
2018-05-05 08:18:51,352 INFO org.apache.zookeeper.ZooKeeper: Session: 0x0 closed
2018-05-05 08:18:51,352 INFO org.apache.helix.manager.zk.ZkClient: Closed zkclient
2018-05-05 08:18:51,352 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 30000
2018-05-05 08:18:51,354 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore failed in state INITED; cause: org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.initializeSubsystem(HBaseTimelineMetricStore.java:118)
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.serviceInit(HBaseTimelineMetricStore.java:96)
.....
The 2188 port is my setting about zookeeper, and from web UI, its status is OK.
Also i check zookeeper process, it runs as usual.
By the way, metrics needs listen 6188 port, but i use `netstat` to check, related processes do not start.
I do not know how to fix it, could someone help me?
Thanks
Re: metrics can not work as expected
Posted by David Quiroga <qu...@gmail.com>.
In most cases Ambari Metrics runs separate HBase and Zookeeper instances.
The default ports of Ambari Metrics Zookeeper are typically in the 60,000
range.
Would expect the Ambari defaults to do the trick, which I suspect relates
to Sid's question.
On Sun, May 6, 2018 at 5:07 AM, <xi...@sky-data.cn> wrote:
> I just download them and make my own repo including them.
> Then, i install them with ambari UI.
>
>
>
> ------------------------------
> *From: *"Siddharth Wagle" <sw...@hortonworks.com>
> *To: *"user" <us...@ambari.apache.org>
> *Sent: *Saturday, May 5, 2018 11:07:12 PM
> *Subject: *Re: metrics can not work as expected
>
> Collector is not able to reach Zookeeper. Are you not installing AMS
> using Ambari?
>
>
> - Sid
>
>
> ------------------------------
> *From:* xiang.dai@sky-data.cn <xi...@sky-data.cn>
> *Sent:* Saturday, May 5, 2018 1:27 AM
> *To:* user
> *Subject:* metrics can not work as expected
>
> Hi!
>
> I tested installation on my vbox, it works well, then i move it to a
> server which run many services.
>
> When install, it failed at check service status.
>
> I checked ambari-metrics-collector.log and found below error:
>
> 2018-05-05 08:16:03,536 INFO org.apache.phoenix.metrics.Metrics:
> Initializing metrics system: phoenix
> 2018-05-05 08:16:03,559 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:
> Cannot locate configuration: tried hadoop-metrics2-phoenix.
> properties,hadoop-metrics2.properties
> ....
>
> 2018-05-05 08:18:50,342 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to dx-1.novalocal/192.168.80.34:2188, initiating
> session
> 2018-05-05 08:18:50,343 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x0, likely server has closed
> socket, closing socket connection and attempting reconnect
> 2018-05-05 08:18:50,811 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server dx-2.novalocal/192.168.80.27:2188. Will not
> attempt to authenticate using SASL (unknown error)
> 2018-05-05 08:18:50,812 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to dx-2.novalocal/192.168.80.27:2188, initiating
> session
> 2018-05-05 08:18:50,813 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x0, likely server has closed
> socket, closing socket connection and attempting reconnect
> ....
>
> 2018-05-05 08:18:50,947 INFO org.apache.helix.manager.zk.ZkClient:
> Closing zkclient: State:CONNECTING sessionid:0x0 local:null
> remoteserver:null lastZxid:0 xid:1 sent:35 recv:0 queuedpkts:0
> pendingresp:0 queuedevents:0
> 2018-05-05 08:18:50,947 INFO org.I0Itec.zkclient.ZkEventThread: Terminate
> ZkClient event thread.
> 2018-05-05 08:18:51,352 INFO org.apache.zookeeper.ZooKeeper: Session: 0x0
> closed
> 2018-05-05 08:18:51,352 INFO org.apache.helix.manager.zk.ZkClient: Closed
> zkclient
> 2018-05-05 08:18:51,352 ERROR org.apache.hadoop.yarn.server.
> applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore:
> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
> zookeeper server within timeout: 30000
> 2018-05-05 08:18:51,354 INFO org.apache.hadoop.service.AbstractService:
> Service org.apache.hadoop.yarn.server.applicationhistoryservice.
> metrics.timeline.HBaseTimelineMetricStore failed in state INITED; cause:
> org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.
> MetricsSystemInitializationException: Unable to initialize HA controller
> org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.
> MetricsSystemInitializationException: Unable to initialize HA controller
> at org.apache.hadoop.yarn.server.applicationhistoryservice.
> metrics.timeline.HBaseTimelineMetricStore.initializeSubsystem(
> HBaseTimelineMetricStore.java:118)
> at org.apache.hadoop.yarn.server.applicationhistoryservice.
> metrics.timeline.HBaseTimelineMetricStore.serviceInit(
> HBaseTimelineMetricStore.java:96)
> .....
>
> The 2188 port is my setting about zookeeper, and from web UI, its status
> is OK.
> Also i check zookeeper process, it runs as usual.
>
> By the way, metrics needs listen 6188 port, but i use `netstat` to check,
> related processes do not start.
>
> I do not know how to fix it, could someone help me?
>
> Thanks
>
Re: metrics can not work as expected
Posted by xi...@sky-data.cn.
I just download them and make my own repo including them.
Then, i install them with ambari UI.
From: "Siddharth Wagle" <sw...@hortonworks.com>
To: "user" <us...@ambari.apache.org>
Sent: Saturday, May 5, 2018 11:07:12 PM
Subject: Re: metrics can not work as expected
Collector is not able to reach Zookeeper. Are you not installing AMS using Ambari?
- Sid
From: xiang.dai@sky-data.cn <xi...@sky-data.cn>
Sent: Saturday, May 5, 2018 1:27 AM
To: user
Subject: metrics can not work as expected
Hi!
I tested installation on my vbox, it works well, then i move it to a server which run many services.
When install, it failed at check service status.
I checked ambari-metrics-collector.log and found below error:
2018-05-05 08:16:03,536 INFO org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix
2018-05-05 08:16:03,559 WARN org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
....
2018-05-05 08:18:50,342 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-1.novalocal/192.168.80.34:2188, initiating session
2018-05-05 08:18:50,343 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-05-05 08:18:50,811 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server dx-2.novalocal/192.168.80.27:2188. Will not attempt to authenticate using SASL (unknown error)
2018-05-05 08:18:50,812 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-2.novalocal/192.168.80.27:2188, initiating session
2018-05-05 08:18:50,813 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
....
2018-05-05 08:18:50,947 INFO org.apache.helix.manager.zk.ZkClient: Closing zkclient: State:CONNECTING sessionid:0x0 local:null remoteserver:null lastZxid:0 xid:1 sent:35 recv:0 queuedpkts:0 pendingresp:0 queuedevents:0
2018-05-05 08:18:50,947 INFO org.I0Itec.zkclient.ZkEventThread: Terminate ZkClient event thread.
2018-05-05 08:18:51,352 INFO org.apache.zookeeper.ZooKeeper: Session: 0x0 closed
2018-05-05 08:18:51,352 INFO org.apache.helix.manager.zk.ZkClient: Closed zkclient
2018-05-05 08:18:51,352 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 30000
2018-05-05 08:18:51,354 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore failed in state INITED; cause: org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.initializeSubsystem(HBaseTimelineMetricStore.java:118)
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.serviceInit(HBaseTimelineMetricStore.java:96)
.....
The 2188 port is my setting about zookeeper, and from web UI, its status is OK.
Also i check zookeeper process, it runs as usual.
By the way, metrics needs listen 6188 port, but i use `netstat` to check, related processes do not start.
I do not know how to fix it, could someone help me?
Thanks
Re: metrics can not work as expected
Posted by Siddharth Wagle <sw...@hortonworks.com>.
?Collector is not able to reach Zookeeper. Are you not installing AMS using Ambari?
- Sid
________________________________
From: xiang.dai@sky-data.cn <xi...@sky-data.cn>
Sent: Saturday, May 5, 2018 1:27 AM
To: user
Subject: metrics can not work as expected
Hi!
I tested installation on my vbox, it works well, then i move it to a server which run many services.
When install, it failed at check service status.
I checked ambari-metrics-collector.log and found below error:
2018-05-05 08:16:03,536 INFO org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix
2018-05-05 08:16:03,559 WARN org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
....
2018-05-05 08:18:50,342 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-1.novalocal/192.168.80.34:2188, initiating session
2018-05-05 08:18:50,343 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-05-05 08:18:50,811 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server dx-2.novalocal/192.168.80.27:2188. Will not attempt to authenticate using SASL (unknown error)
2018-05-05 08:18:50,812 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dx-2.novalocal/192.168.80.27:2188, initiating session
2018-05-05 08:18:50,813 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
....
2018-05-05 08:18:50,947 INFO org.apache.helix.manager.zk.ZkClient: Closing zkclient: State:CONNECTING sessionid:0x0 local:null remoteserver:null lastZxid:0 xid:1 sent:35 recv:0 queuedpkts:0 pendingresp:0 queuedevents:0
2018-05-05 08:18:50,947 INFO org.I0Itec.zkclient.ZkEventThread: Terminate ZkClient event thread.
2018-05-05 08:18:51,352 INFO org.apache.zookeeper.ZooKeeper: Session: 0x0 closed
2018-05-05 08:18:51,352 INFO org.apache.helix.manager.zk.ZkClient: Closed zkclient
2018-05-05 08:18:51,352 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 30000
2018-05-05 08:18:51,354 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore failed in state INITED; cause: org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Unable to initialize HA controller
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.initializeSubsystem(HBaseTimelineMetricStore.java:118)
at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.serviceInit(HBaseTimelineMetricStore.java:96)
.....
The 2188 port is my setting about zookeeper, and from web UI, its status is OK.
Also i check zookeeper process, it runs as usual.
By the way, metrics needs listen 6188 port, but i use `netstat` to check, related processes do not start.
I do not know how to fix it, could someone help me?
Thanks