You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by James Ram <hb...@gmail.com> on 2011/05/25 14:07:08 UTC
ClockOutOfSyncException with zookeeper
Hi,
I am using 1 master and 4 slave machines cluster. I tried to add three
HRegionserver including the master. But its throwing the following
exception. The HRegionserver is running on the master but its not running on
the slaves. I have also given the hbase-site.xml and regionservers also.
Please help me to find out the problem.
*hbase-site.xml*
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://newcloud/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>160.110.185.93,160.110.79.33,160.110.79.60,160.110.185.160,160.110.185.183</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/tmp/hbase-sao_user/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.leaderport</name>
<value>3888</value>
<description>Port used by ZooKeeper for leader election.
See
http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper
for more information.
</description>
</property>
</configuration>
*regionservers*
160.110.185.93
160.110.79.33
160.110.79.60
org.apache.hadoop.hbase.ClockOutOfSyncException:
org.apache.hadoop.hbase.ClockOutOfSyncException: Server
160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
far out of sync with master. Time difference of 19832277ms > max allowed of
30000ms^M
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)^M
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)^M
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)^M
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)^M
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)^M
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)^M
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1504)^M
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1468)^M
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:563)^M
at java.lang.Thread.run(Thread.java:619)^M
Caused by: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.ClockOutOfSyncException: Server
160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
far out of sync with master. Time difference of 19832277ms > max allowed of
30000ms^M
at
org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:181)^M
at
org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:129)^M
at
org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:612)^M
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)^M
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)^M
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)^M
at java.lang.reflect.Method.invoke(Unknown Source)^M
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)^M
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)^M
^M
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:753)^M
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)^M
at $Proxy2.regionServerStartup(Unknown Source)^M
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1500)^M
... 3 more^M
--
With Regards,
Jr.
Re: ClockOutOfSyncException with zookeeper
Posted by James Ram <hb...@gmail.com>.
Hi Harsh / Christian,
Thankyou for your reply. The issue is solved. There was a difference between
master and 2 slaves(Regionservers).
On Wed, May 25, 2011 at 5:41 PM, Kleegrewe, Christian <
christian.kleegrewe@siemens.com> wrote:
> Hi Jr.
>
> Have a look at the Exception message:
>
> org.apache.hadoop.hbase.ClockOutOfSyncException: Server
> 160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
> far out of sync with master. Time difference of 19832277ms > max allowed
> of
> 30000ms
>
>
> It seems that the system time of the server you like to add to the cluster
> differs too much from the system time of the master. System times of both
> machines are allowed to differ 30 seconds. To overcome the problem change
> the system time of the new region server to your masters system time.
>
> regards
>
> Christian
>
> ------------------8<--------------------------
>
> Siemens AG
> Corporate Technology
> Corporate Research and Technologies
> CT T DE IT3
> Otto-Hahn-Ring 6
> 81739 München, Deutschland
> Tel.: +49 (89) 636-42722
> Fax: +49 (89) 636-41423
> mailto:christian.kleegrewe@siemens.com
>
> Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme;
> Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus
> Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm,
> Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München,
> Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB
> 6684; WEEE-Reg.-Nr. DE 23691322
>
>
> -----Original Message-----
> From: James Ram [mailto:hbaserj@gmail.com]
> Sent: Wednesday, May 25, 2011 2:07 PM
> To: user@hbase.apache.org
> Subject: ClockOutOfSyncException with zookeeper
>
> Hi,
>
> I am using 1 master and 4 slave machines cluster. I tried to add three
> HRegionserver including the master. But its throwing the following
> exception. The HRegionserver is running on the master but its not running
> on
> the slaves. I have also given the hbase-site.xml and regionservers also.
> Please help me to find out the problem.
>
> *hbase-site.xml*
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://newcloud/hbase</value>
> </property>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>2181</value>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
>
>
> <value>160.110.185.93,160.110.79.33,160.110.79.60,160.110.185.160,160.110.185.183</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/tmp/hbase-sao_user/zookeeper</value>
> </property>
> <property>
> <name>hbase.zookeeper.leaderport</name>
> <value>3888</value>
> <description>Port used by ZooKeeper for leader election.
> See
>
> http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper
> for more information.
> </description>
> </property>
> </configuration>
>
> *regionservers*
>
> 160.110.185.93
> 160.110.79.33
> 160.110.79.60
>
> org.apache.hadoop.hbase.ClockOutOfSyncException:
> org.apache.hadoop.hbase.ClockOutOfSyncException: Server
> 160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
> far out of sync with master. Time difference of 19832277ms > max allowed
> of
> 30000ms^M
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)^M
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)^M
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)^M
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)^M
> at
>
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)^M
> at
>
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)^M
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1504)^M
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1468)^M
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:563)^M
> at java.lang.Thread.run(Thread.java:619)^M
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.ClockOutOfSyncException: Server
> 160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
> far out of sync with master. Time difference of 19832277ms > max allowed
> of
> 30000ms^M
> at
>
> org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:181)^M
> at
>
> org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:129)^M
> at
>
> org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:612)^M
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)^M
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)^M
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)^M
> at java.lang.reflect.Method.invoke(Unknown Source)^M
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)^M
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)^M
> ^M
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:753)^M
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)^M
> at $Proxy2.regionServerStartup(Unknown Source)^M
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1500)^M
> ... 3 more^M
>
>
> --
> With Regards,
> Jr.
>
--
With Regards,
Jr.
RE: ClockOutOfSyncException with zookeeper
Posted by "Kleegrewe, Christian" <ch...@siemens.com>.
Hi Jr.
Have a look at the Exception message:
org.apache.hadoop.hbase.ClockOutOfSyncException: Server
160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
far out of sync with master. Time difference of 19832277ms > max allowed of
30000ms
It seems that the system time of the server you like to add to the cluster differs too much from the system time of the master. System times of both machines are allowed to differ 30 seconds. To overcome the problem change the system time of the new region server to your masters system time.
regards
Christian
------------------8<--------------------------
Siemens AG
Corporate Technology
Corporate Research and Technologies
CT T DE IT3
Otto-Hahn-Ring 6
81739 München, Deutschland
Tel.: +49 (89) 636-42722
Fax: +49 (89) 636-41423
mailto:christian.kleegrewe@siemens.com
Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme; Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
-----Original Message-----
From: James Ram [mailto:hbaserj@gmail.com]
Sent: Wednesday, May 25, 2011 2:07 PM
To: user@hbase.apache.org
Subject: ClockOutOfSyncException with zookeeper
Hi,
I am using 1 master and 4 slave machines cluster. I tried to add three
HRegionserver including the master. But its throwing the following
exception. The HRegionserver is running on the master but its not running on
the slaves. I have also given the hbase-site.xml and regionservers also.
Please help me to find out the problem.
*hbase-site.xml*
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://newcloud/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>160.110.185.93,160.110.79.33,160.110.79.60,160.110.185.160,160.110.185.183</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/tmp/hbase-sao_user/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.leaderport</name>
<value>3888</value>
<description>Port used by ZooKeeper for leader election.
See
http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper
for more information.
</description>
</property>
</configuration>
*regionservers*
160.110.185.93
160.110.79.33
160.110.79.60
org.apache.hadoop.hbase.ClockOutOfSyncException:
org.apache.hadoop.hbase.ClockOutOfSyncException: Server
160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
far out of sync with master. Time difference of 19832277ms > max allowed of
30000ms^M
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)^M
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)^M
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)^M
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)^M
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)^M
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)^M
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1504)^M
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1468)^M
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:563)^M
at java.lang.Thread.run(Thread.java:619)^M
Caused by: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.ClockOutOfSyncException: Server
160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
far out of sync with master. Time difference of 19832277ms > max allowed of
30000ms^M
at
org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:181)^M
at
org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:129)^M
at
org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:612)^M
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)^M
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)^M
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)^M
at java.lang.reflect.Method.invoke(Unknown Source)^M
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)^M
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)^M
^M
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:753)^M
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)^M
at $Proxy2.regionServerStartup(Unknown Source)^M
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1500)^M
... 3 more^M
--
With Regards,
Jr.
Re: ClockOutOfSyncException with zookeeper
Posted by Harsh J <ha...@cloudera.com>.
Here's an initial thought question: Is your cluster nodes' time in
sync with one another? i.e., is ntpd functional on each? HBase/ZK
would require clocks on each node to be synchronized with one another.
On Wed, May 25, 2011 at 5:37 PM, James Ram <hb...@gmail.com> wrote:
> Hi,
>
> I am using 1 master and 4 slave machines cluster. I tried to add three
> HRegionserver including the master. But its throwing the following
> exception. The HRegionserver is running on the master but its not running on
> the slaves. I have also given the hbase-site.xml and regionservers also.
> Please help me to find out the problem.
>
> *hbase-site.xml*
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://newcloud/hbase</value>
> </property>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>2181</value>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
>
> <value>160.110.185.93,160.110.79.33,160.110.79.60,160.110.185.160,160.110.185.183</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/tmp/hbase-sao_user/zookeeper</value>
> </property>
> <property>
> <name>hbase.zookeeper.leaderport</name>
> <value>3888</value>
> <description>Port used by ZooKeeper for leader election.
> See
> http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperStarted.html#sc_RunningReplicatedZooKeeper
> for more information.
> </description>
> </property>
> </configuration>
>
> *regionservers*
>
> 160.110.185.93
> 160.110.79.33
> 160.110.79.60
>
> org.apache.hadoop.hbase.ClockOutOfSyncException:
> org.apache.hadoop.hbase.ClockOutOfSyncException: Server
> 160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
> far out of sync with master. Time difference of 19832277ms > max allowed of
> 30000ms^M
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)^M
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)^M
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)^M
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)^M
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)^M
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)^M
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1504)^M
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1468)^M
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:563)^M
> at java.lang.Thread.run(Thread.java:619)^M
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.ClockOutOfSyncException: Server
> 160.110.79.60,60020,1306320747968 has been rejected; Reported time is too
> far out of sync with master. Time difference of 19832277ms > max allowed of
> 30000ms^M
> at
> org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:181)^M
> at
> org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:129)^M
> at
> org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:612)^M
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)^M
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)^M
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)^M
> at java.lang.reflect.Method.invoke(Unknown Source)^M
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)^M
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)^M
> ^M
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:753)^M
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)^M
> at $Proxy2.regionServerStartup(Unknown Source)^M
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1500)^M
> ... 3 more^M
>
>
> --
> With Regards,
> Jr.
>
--
Harsh J