You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Eirik Thorsnes <ei...@uni.no> on 2016/04/18 14:23:34 UTC
Ambari Metrics gives NumberFormatException and dies
Hi All,
I get the below exception in the hbase-ams-master and
ambari-metrics-collector logs. Repeating 35 times with the same input
string "7.0" until the collector dies. Looks like a float-int mismatch.
Any suggestions how I get it to "move on"?
Regards,
Eirik
# ambari-metrics-collector log:
2016-04-18 14:16:53,069 INFO [main] RpcRetryingCaller:132 - Call
exception, tries=34, retries=35, started=531246 ms ago, cancelled=false,
msg=
2016-04-18 14:16:53,082 INFO [main]
ConnectionManager$HConnectionImplementation:2093 - Closing master
protocol: MasterService
2016-04-18 14:16:53,083 INFO [main]
ConnectionManager$HConnectionImplementation:1682 - Closing zookeeper
sessionid=0x253c35a7d362b49
2016-04-18 14:16:53,085 INFO [main] ZooKeeper:684 - Session:
0x253c35a7d362b49 closed
2016-04-18 14:16:53,085 INFO [main-EventThread] ClientCnxn:524 -
EventThread shut down
2016-04-18 14:16:53,096 ERROR [main] PhoenixHBaseAccessor:321 - Error
creating Metrics Schema in HBase using Phoenix.
org.apache.phoenix.exception.PhoenixIOException: Failed after
attempts=35, exceptions:
Mon Apr 18 14:08:01 CEST 2016,
RpcRetryingCaller{globalStartTime=1460981281823, pause=100, retries=35},
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
java.io.IOException: For input string: "7.0"
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2156)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NumberFormatException: For input string: "7.0"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.valueOf(Integer.java:766)
at
org.apache.hadoop.hbase.HColumnDescriptor.getTimeToLive(HColumnDescriptor.java:854)
at
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1630)
at
org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2091)
at
org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1125)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51095)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
... 4 more
# hbase-ams-master logs:
2016-04-18 14:14:32,202 ERROR
[B.defaultRpcServer.handler=16,queue=1,port=61300] ipc.RpcServer:
Unexpected throwable object
java.lang.NumberFormatException: For input string: "7.0"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.valueOf(Integer.java:766)
at
org.apache.hadoop.hbase.HColumnDescriptor.getTimeToLive(HColumnDescriptor.java:854)
at
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1630)
at
org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2091)
at
org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1125)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51095)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
--
Eirik Thorsnes
Re: Ambari Metrics gives NumberFormatException and dies
Posted by Eirik Thorsnes <ei...@uni.no>.
On 18. april 2016 19:52, Aravindan Vijayan wrote:
> Hi Eirik,
>
> This is most likely because of the bug that existed between the fixes
> for AMBARI-14982 and AMBARI-15402. Can you verify if the values of
> the following ams-site Hbase TTL configs are consistent with those on
> your cluster? If no, please go ahead and change them accordingly.
> Then restart AMS collector.
>
>
Hi Aravindan,
My values for your listed timeline.metric.*.ttl parameters were very
different, much lower. Changing values and restarting AMS fixed the issue.
Thank you,
Eirik
--
Eirik Thorsnes
Re: Ambari Metrics gives NumberFormatException and dies
Posted by Aravindan Vijayan <av...@hortonworks.com>.
Hi Eirik,
This is most likely because of the bug that existed between the fixes for AMBARI-14982 and AMBARI-15402. Can you verify if the values of the following ams-site Hbase TTL configs are consistent with those on your cluster? If no, please go ahead and change them accordingly. Then restart AMS collector.
--
Thanks and Regards,
Aravindan Vijayan
On 4/18/16, 9:35 AM, "Eirik Thorsnes" <ei...@uni.no> wrote:
>
>
>On 18.04.2016 16.27, Aravindan Vijayan wrote:
>> Hi Eirik,
>>
>> What version and exact build number of Ambari is this? Did you do an upgrade recently?
>>
>
>Hi Aravindan,
>
>Yes, I did an upgrade recently. Due to some instabilities on Metrics I
>wanted to try the newest git (which looking back clearly made things
>worse). Upgrade sequence as follows:
>
>1. on 2.2.1
>2. upgrade to git snapshot from branch-2.2 on Feb 16 (680502c)
>3. upgrade to git snapshot from branch-2.2 on Mar 30 (a9e65de)
>4. downgrade to point 2. (680502c) on Mar 30
>5. upgrade to git snapshot from branch-2.2 on Apr 18 (b40ae34)
>
>(Numbers in paranthesis are latest commit for the branch). Issue occured
>at step 3. or 4. above. I tried today to upgrade to latest on branch-2.2
>but problem persists.
>
>It took some time to spot the issue due to the auto-restart feature of
>Metrics. The downgrade in point 4. was due to other issues preventing
>ambari-server to work.
>
>Regards,
>
>Eirik
>
Re: Ambari Metrics gives NumberFormatException and dies
Posted by Eirik Thorsnes <ei...@uni.no>.
On 18.04.2016 16.27, Aravindan Vijayan wrote:
> Hi Eirik,
>
> What version and exact build number of Ambari is this? Did you do an upgrade recently?
>
Hi Aravindan,
Yes, I did an upgrade recently. Due to some instabilities on Metrics I
wanted to try the newest git (which looking back clearly made things
worse). Upgrade sequence as follows:
1. on 2.2.1
2. upgrade to git snapshot from branch-2.2 on Feb 16 (680502c)
3. upgrade to git snapshot from branch-2.2 on Mar 30 (a9e65de)
4. downgrade to point 2. (680502c) on Mar 30
5. upgrade to git snapshot from branch-2.2 on Apr 18 (b40ae34)
(Numbers in paranthesis are latest commit for the branch). Issue occured
at step 3. or 4. above. I tried today to upgrade to latest on branch-2.2
but problem persists.
It took some time to spot the issue due to the auto-restart feature of
Metrics. The downgrade in point 4. was due to other issues preventing
ambari-server to work.
Regards,
Eirik
Re: Ambari Metrics gives NumberFormatException and dies
Posted by Aravindan Vijayan <av...@hortonworks.com>.
Hi Eirik,
What version and exact build number of Ambari is this? Did you do an upgrade recently?
--
Thanks and Regards,
Aravindan Vijayan
On 4/18/16, 5:23 AM, "Eirik Thorsnes" <ei...@uni.no> wrote:
>Hi All,
>
>I get the below exception in the hbase-ams-master and
>ambari-metrics-collector logs. Repeating 35 times with the same input
>string "7.0" until the collector dies. Looks like a float-int mismatch.
>Any suggestions how I get it to "move on"?
>
>Regards,
>Eirik
>
># ambari-metrics-collector log:
>
>2016-04-18 14:16:53,069 INFO [main] RpcRetryingCaller:132 - Call
>exception, tries=34, retries=35, started=531246 ms ago, cancelled=false,
>msg=
>2016-04-18 14:16:53,082 INFO [main]
>ConnectionManager$HConnectionImplementation:2093 - Closing master
>protocol: MasterService
>2016-04-18 14:16:53,083 INFO [main]
>ConnectionManager$HConnectionImplementation:1682 - Closing zookeeper
>sessionid=0x253c35a7d362b49
>2016-04-18 14:16:53,085 INFO [main] ZooKeeper:684 - Session:
>0x253c35a7d362b49 closed
>2016-04-18 14:16:53,085 INFO [main-EventThread] ClientCnxn:524 -
>EventThread shut down
>2016-04-18 14:16:53,096 ERROR [main] PhoenixHBaseAccessor:321 - Error
>creating Metrics Schema in HBase using Phoenix.
>org.apache.phoenix.exception.PhoenixIOException: Failed after
>attempts=35, exceptions:
>Mon Apr 18 14:08:01 CEST 2016,
>RpcRetryingCaller{globalStartTime=1460981281823, pause=100, retries=35},
>org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
>java.io.IOException: For input string: "7.0"
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2156)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
>org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at
>org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
>Caused by: java.lang.NumberFormatException: For input string: "7.0"
> at
>java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580)
> at java.lang.Integer.valueOf(Integer.java:766)
> at
>org.apache.hadoop.hbase.HColumnDescriptor.getTimeToLive(HColumnDescriptor.java:854)
> at
>org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1630)
> at
>org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2091)
> at
>org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1125)
> at
>org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51095)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> ... 4 more
>
>
>
># hbase-ams-master logs:
>
>2016-04-18 14:14:32,202 ERROR
>[B.defaultRpcServer.handler=16,queue=1,port=61300] ipc.RpcServer:
>Unexpected throwable object
>java.lang.NumberFormatException: For input string: "7.0"
> at
>java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580)
> at java.lang.Integer.valueOf(Integer.java:766)
> at
>org.apache.hadoop.hbase.HColumnDescriptor.getTimeToLive(HColumnDescriptor.java:854)
> at
>org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1630)
> at
>org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2091)
> at
>org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1125)
> at
>org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51095)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
>org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at
>org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
>
>--
>Eirik Thorsnes
>
>