You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Siddharth Wagle (JIRA)" <ji...@apache.org> on 2015/11/03 00:00:28 UTC
[jira] [Commented] (AMBARI-13684) Ambari Metrics Collector failed
to start in Ambari 2.1.2 with UnicodeEncodeError
[ https://issues.apache.org/jira/browse/AMBARI-13684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14986218#comment-14986218 ]
Siddharth Wagle commented on AMBARI-13684:
------------------------------------------
The problem is with special characters in the KAFKA service metrics file. We use this file as a part of stack_advisor calculations for AMS split points in 2.1.2
Following is a grep on non ASCII characters that reveals the problem.
{noformat}
root@os-u12-upgrade-2ga-latest-4:~# grep --color='auto' -P -n "[\x80-\xFF]" /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/package/files/service-metrics/KAFKA.txt
43:���kafka.network.RequestMetrics.RequestsPerSec.request.OffsetFetch.count���
45:���kafka.network.RequestMetrics.RequestsPerSec.request.OffsetCommit.count
47:kafka.network.RequestMetrics.RequestsPerSec.request.LeaderAndIsr.1MinuteRate���
49:���kafka.network.RequestMetrics.RequestsPerSec.request.JoinGroup.meanRate���
51:���kafka.network.RequestMetrics.RequestsPerSec.request.JoinGroup.5MinuteRate���
52:kafka.network.RequestMetrics.RequestsPerSec.request.JoinGroup.1MinuteRate���
53:kafka.network.RequestMetrics.RequestsPerSec.request.JoinGroup.15MinuteRate���
55:���kafka.network.RequestMetrics.RequestsPerSec.request.Heartbeat.count
56:���kafka.network.RequestMetrics.RequestsPerSec.request.Heartbeat.5MinuteRate
57:���kafka.network.RequestMetrics.RequestsPerSec.request.Heartbeat.1MinuteRate���
58:kafka.network.RequestMetrics.RequestsPerSec.request.Heartbeat.15MinuteRate���
60:kafka.network.RequestMetrics.RequestsPerSec.request.FetchFollower.5MinuteRate���
62:���kafka.network.RequestMetrics.RequestsPerSec.request.FetchFollower.15MinuteRate
63:���kafka.network.RequestMetrics.RequestsPerSec.request.FetchConsumer.meanRate
64:���kafka.network.RequestMetrics.RequestsPerSec.request.FetchConsumer.count���
66:���kafka.network.RequestMetrics.RequestsPerSec.request.FetchConsumer.1MinuteRate
67:���kafka.network.RequestMetrics.RequestsPerSec.request.Fetch.count
{noformat}
Other service metrics are ok.
*Why is this not always reproducible*:
- Split point calculation picks up 50 equidistant points for each application to use for the final split points result.
- We have about 17 points that are bad out of 190, so we have a 4.47 % chance of picking a bad point.
- User needs to have Kafka installed
*Workaround*:
- Call API to get site configuration for ams:
{code}
http://172.22.64.164:8080/api/v1/clusters/cl1/configurations?type=ams-site
{code}
Pick latest
- Check the value of _timeline.metrics.cluster.aggregate.splitpoints_ and _timeline.metrics.host.aggregate.splitpoints_
- Look for special non-ascii characters example:
{code}
"dfs.datanode.ReplaceBlockOpAvgTime,kafka.network.RequestMetrics.RequestsPerSec.request.JoinGroup.1MinuteRate
,master.Master.ProcessCallTime_num_ops,regionserver.Server.blockCacheEvictionCount"
{code}
Check _1MinuteRate_ in the example above.
- Use _/var/lib/ambari-server/resources/scripts/configs.sh_ to modify the value and remove the special character and restart the service.
> Ambari Metrics Collector failed to start in Ambari 2.1.2 with UnicodeEncodeError
> --------------------------------------------------------------------------------
>
> Key: AMBARI-13684
> URL: https://issues.apache.org/jira/browse/AMBARI-13684
> Project: Ambari
> Issue Type: Bug
> Components: ambari-metrics, ambari-server
> Affects Versions: 2.1.2
> Reporter: Siddharth Wagle
> Assignee: Siddharth Wagle
> Priority: Critical
> Fix For: 2.1.3
>
>
> Stderr
> {noformat}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 131, in <module>
> AmsCollector().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 216, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 44, in start
> self.configure(env, action = 'start') # for security
> File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 41, in configure
> ams(name='collector')
> File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
> return fn(*args, **kwargs)
> File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py", line 202, in ams
> group=params.user_group
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/xml_config.py", line 67, in action_create
> encoding = self.resource.encoding
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 90, in action_create
> content = self._get_content()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 127, in _get_content
> return content()
> File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__
> return self.get_content()
> File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 142, in get_content
> rendered = self.template.render(self.context)
> File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render
> return self.environment.handle_exception(exc_info, True)
> File "<template>", line 5, in top-level template code
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u2028' in position 108: ordinal not in range(128)
> {noformat}
> |Old Ambari|NEW Ambari|HDP Stack
> |Ambari 1.7.0|Ambari2.1.2|2.2.0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)