You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Sid Wagle <sw...@hortonworks.com> on 2017/03/27 22:12:29 UTC
Re: Review Request 57902: AMBARI-20553 : Ambari script error for
ams-hbase while writing to Amazon s3 on a cluster with no HDFS.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57902/#review170231
-----------------------------------------------------------
Missing UpgradeCatalog changes.
- Sid Wagle
On None, Aravindan Vijayan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57902/
> -----------------------------------------------------------
>
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20553
> https://issues.apache.org/jira/browse/AMBARI-20553
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Metrics collector startup scripts fail to handle s3 paths in configurations:
>
> {
> "hbase-site":{
> "properties":{
> "hbase.rootdir":"s3a://ss-datasets/apps/hbase/",
> "hbase.wal.dir":"file:///usr/lib/ams-hbase/data"
> }
> }
>
> Exception Trace
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 150, in <module>
> AmsCollector().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 313, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 766, in restart
> self.start(env)
> File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 48, in start
> self.configure(env, action = 'start') # for security
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 116, in locking_configure
> original_configure(obj, *args, **kw)
> File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 43, in configure
> hbase('master', action)
> File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
> return fn(*args, **kwargs)
> File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py", line 213, in hbase
> dfs_type=params.dfs_type
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 555, in action_create_on_execute
> self.action_delayed("create")
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 552, in action_delayed
> self.get_hdfs_resource_executor().action_delayed(action_name, self)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 279, in action_delayed
> self._assert_valid()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 238, in _assert_valid
> self.target_status = self._get_file_status(target)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 381, in _get_file_status
> list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 186, in run_command
> _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output
> raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
> resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://<host>:50070/webhdfs/v1s3a:/ss-datasets/apps/hbase?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpjkn3uB 2>/tmp/tmpCVb8Kl' returned 7. curl: (7) Failed to connect to <host> port 50070: Connection refused
> 000
>
>
> FIX
> 1. Made HBase rootdir creation on remote file system optional.
> 2. Added ability to specify wal dir for AMS HBase.
> 3. Added ability to specify additional classpath artifacts for Metrics Collector.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-timelineservice/conf/unix/ambari-metrics-collector c106235
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml 81fa4c7
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py 26b3502
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py 3276cc1
>
>
> Diff: https://reviews.apache.org/r/57902/diff/1/
>
>
> Testing
> -------
>
> Manually tested.
>
> Python unit tests pass.
>
>
> Thanks,
>
> Aravindan Vijayan
>
>