You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Andrew Onischuk <ao...@hortonworks.com> on 2015/02/25 16:45:19 UTC
Review Request 31414: [Monarch] Cluster create failed with timeout on
the client side
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31414/
-----------------------------------------------------------
Review request for Ambari and Dmitro Lisnichenko.
Bugs: AMBARI-9795
https://issues.apache.org/jira/browse/AMBARI-9795
Repository: ambari
Description
-------
We saw one cluster create fail in Production with the below error during
Ambari install:
Fail: Execution of 'apt-get update -qq -o Dir::Etc::sourcelist=sources.list.d/HDP.list -o APT::Get::List-Cleanup=0' returned 100. E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock directory /var/lib/apt/lists/"
Detailed log:
{
"href" : "http://headnode0.mjlinux15-ssh.c3.internal.cloudapp.net:8080/api/v1/clusters/mjlinux15/requests/1/tasks/27",
"Tasks" : {
"cluster_name" : "mjlinux15",
"command_detail" : "DATANODE INSTALL",
"id" : 27,
"request_id" : 1,
"stage_id" : 1,
"status" : "FAILED",
"stderr" : "2015-02-18 06:55:16,758 - Error while executing command 'install':\nTraceback (most recent call last):\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 184, in execute\n method(env)\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/hook.py\", line 33, in hook\n install_repos()\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\", line 55, in install_repos\n _alter_repo(\"create\", params.repo_info, template)\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\", line 49, in _alter_repo\n components = ubuntu_components, # ubuntu specific\n File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 148, in __init__\n self.env.run()\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 151, in run\n
self.run_action(resource, action)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 117, in run_action\n provider_action()\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/repository.py\", line 97, in action_create\n retcode, out = checked_call(update_cmd_formatted, sudo=True)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 67, in inner\n return function(command, **kwargs)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 79, in checked_call\n return _call(command, logoutput, True, cwd, env, preexec_fn, user, wait_for_finish, timeout, path, sudo, on_new_line)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 185, in _call\n raise Fail(err_msg)\nFail: Execution of 'apt-get update -qq -o Dir::Etc::sourcelist=sources.list.d/HDP.list -o APT::Get::List-Cleanup=0' returned 100. E: Could not
get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock directory /var/lib/apt/lists/"
}
},
{
"href" : "http://headnode0.mjlinux15-ssh.c3.internal.cloudapp.net:8080/api/v1/clusters/mjlinux15/requests/1/tasks/28",
"Tasks" : {
"cluster_name" : "mjlinux15",
"command_detail" : "METRIC_MONITOR INSTALL",
"id" : 28,
"request_id" : 1,
"stage_id" : 1,
"status" : "FAILED",
"stderr" : "2015-02-18 06:55:18,118 - Error while executing command 'install':\nTraceback (most recent call last):\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 184, in execute\n method(env)\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/hook.py\", line 33, in hook\n install_repos()\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\", line 55, in install_repos\n _alter_repo(\"create\", params.repo_info, template)\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\", line 49, in _alter_repo\n components = ubuntu_components, # ubuntu specific\n File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 148, in __init__\n self.env.run()\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 151, in run\n
self.run_action(resource, action)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 117, in run_action\n provider_action()\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/repository.py\", line 97, in action_create\n retcode, out = checked_call(update_cmd_formatted, sudo=True)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 67, in inner\n return function(command, **kwargs)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 79, in checked_call\n return _call(command, logoutput, True, cwd, env, preexec_fn, user, wait_for_finish, timeout, path, sudo, on_new_line)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 185, in _call\n raise Fail(err_msg)\nFail: Execution of 'apt-get update -qq -o Dir::Etc::sourcelist=sources.list.d/HDP-UTILS.list -o APT::Get::List-Cleanup=0' returned 100. E: Coul
d not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock directory /var/lib/apt/lists/"
}
},
This is being seen on Azure with Ubuntu and we need to find out why this is
happening and see if this is an OS issue or Ambari issue.
Diffs
-----
ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py bf24730
ambari-agent/src/main/python/ambari_agent/HostCleanup.py 398502e
ambari-agent/src/main/python/ambari_agent/HostInfo.py dc0834a
ambari-agent/src/test/python/ambari_agent/TestHostCheckReportFileHandler.py eb7efb2
ambari-agent/src/test/python/ambari_agent/TestHostCleanup.py f43784c
ambari-agent/src/test/python/ambari_agent/TestHostInfo.py c1b93d6
ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py 888387c
ambari-server/src/main/resources/custom_actions/scripts/check_host.py ff2eabd
ambari-server/src/test/python/custom_actions/TestCheckHost.py 105a0b0
Diff: https://reviews.apache.org/r/31414/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 31414: [Monarch] Cluster create failed with
timeout on the client side
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31414/#review74032
-----------------------------------------------------------
Ship it!
Ship It!
- Dmitro Lisnichenko
On Feb. 25, 2015, 3:45 p.m., Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31414/
> -----------------------------------------------------------
>
> (Updated Feb. 25, 2015, 3:45 p.m.)
>
>
> Review request for Ambari and Dmitro Lisnichenko.
>
>
> Bugs: AMBARI-9795
> https://issues.apache.org/jira/browse/AMBARI-9795
>
>
> Repository: ambari
>
>
> Description
> -------
>
> We saw one cluster create fail in Production with the below error during
> Ambari install:
>
>
>
>
> Fail: Execution of 'apt-get update -qq -o Dir::Etc::sourcelist=sources.list.d/HDP.list -o APT::Get::List-Cleanup=0' returned 100. E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock directory /var/lib/apt/lists/"
>
>
> Detailed log:
>
>
>
>
> {
> "href" : "http://headnode0.mjlinux15-ssh.c3.internal.cloudapp.net:8080/api/v1/clusters/mjlinux15/requests/1/tasks/27",
> "Tasks" : {
> "cluster_name" : "mjlinux15",
> "command_detail" : "DATANODE INSTALL",
> "id" : 27,
> "request_id" : 1,
> "stage_id" : 1,
> "status" : "FAILED",
> "stderr" : "2015-02-18 06:55:16,758 - Error while executing command 'install':\nTraceback (most recent call last):\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 184, in execute\n method(env)\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/hook.py\", line 33, in hook\n install_repos()\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\", line 55, in install_repos\n _alter_repo(\"create\", params.repo_info, template)\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\", line 49, in _alter_repo\n components = ubuntu_components, # ubuntu specific\n File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 148, in __init__\n self.env.run()\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 151, in run
\n self.run_action(resource, action)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 117, in run_action\n provider_action()\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/repository.py\", line 97, in action_create\n retcode, out = checked_call(update_cmd_formatted, sudo=True)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 67, in inner\n return function(command, **kwargs)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 79, in checked_call\n return _call(command, logoutput, True, cwd, env, preexec_fn, user, wait_for_finish, timeout, path, sudo, on_new_line)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 185, in _call\n raise Fail(err_msg)\nFail: Execution of 'apt-get update -qq -o Dir::Etc::sourcelist=sources.list.d/HDP.list -o APT::Get::List-Cleanup=0' returned 100. E: Could no
t get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock directory /var/lib/apt/lists/"
> }
> },
> {
> "href" : "http://headnode0.mjlinux15-ssh.c3.internal.cloudapp.net:8080/api/v1/clusters/mjlinux15/requests/1/tasks/28",
> "Tasks" : {
> "cluster_name" : "mjlinux15",
> "command_detail" : "METRIC_MONITOR INSTALL",
> "id" : 28,
> "request_id" : 1,
> "stage_id" : 1,
> "status" : "FAILED",
> "stderr" : "2015-02-18 06:55:18,118 - Error while executing command 'install':\nTraceback (most recent call last):\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 184, in execute\n method(env)\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/hook.py\", line 33, in hook\n install_repos()\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\", line 55, in install_repos\n _alter_repo(\"create\", params.repo_info, template)\n File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py\", line 49, in _alter_repo\n components = ubuntu_components, # ubuntu specific\n File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 148, in __init__\n self.env.run()\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 151, in run
\n self.run_action(resource, action)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 117, in run_action\n provider_action()\n File \"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/repository.py\", line 97, in action_create\n retcode, out = checked_call(update_cmd_formatted, sudo=True)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 67, in inner\n return function(command, **kwargs)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 79, in checked_call\n return _call(command, logoutput, True, cwd, env, preexec_fn, user, wait_for_finish, timeout, path, sudo, on_new_line)\n File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 185, in _call\n raise Fail(err_msg)\nFail: Execution of 'apt-get update -qq -o Dir::Etc::sourcelist=sources.list.d/HDP-UTILS.list -o APT::Get::List-Cleanup=0' returned 100. E: Co
uld not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock directory /var/lib/apt/lists/"
> }
> },
>
>
> This is being seen on Azure with Ubuntu and we need to find out why this is
> happening and see if this is an OS issue or Ambari issue.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py bf24730
> ambari-agent/src/main/python/ambari_agent/HostCleanup.py 398502e
> ambari-agent/src/main/python/ambari_agent/HostInfo.py dc0834a
> ambari-agent/src/test/python/ambari_agent/TestHostCheckReportFileHandler.py eb7efb2
> ambari-agent/src/test/python/ambari_agent/TestHostCleanup.py f43784c
> ambari-agent/src/test/python/ambari_agent/TestHostInfo.py c1b93d6
> ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py 888387c
> ambari-server/src/main/resources/custom_actions/scripts/check_host.py ff2eabd
> ambari-server/src/test/python/custom_actions/TestCheckHost.py 105a0b0
>
> Diff: https://reviews.apache.org/r/31414/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>