You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Konrad Hübner <ko...@gmail.com> on 2015/08/20 09:20:04 UTC
Fwd: Issue with deploying Cluster through Ambari Agent behind proxy
Hi,
I try to deploy a HDP 2.3 Cluster with Ambari 2.1. By force I am behind a
proxy server. The proxy is configured both systemwide in /etc/env,
including export of environment variables for sudoers and in
/etc/apt/apt.conf.d with Accquire::proxy command. Ambari Server was
configured to use a non-root users with sudo privilege. Installing ambari
client worked, but when starting the deployment of the cluster through the
agent, the agent runs apt-get install and that failes when resolving the
hostname. This looks like it is not using the proxy. Please find the log
output below. Any help is appreciated!
Thanks
-Konrad
2015-08-20 06:53:13,299 - Package['unzip'] {}
2015-08-20 06:53:13,310 - Installing package unzip ('/usr/bin/apt-get
-q -o Dpkg::Options::=--force-confdef --allow-unauthenticated
--assume-yes install unzip')
2015-08-20 06:53:13,514 - Execution of '['/usr/bin/apt-get', '-q',
'-o', 'Dpkg::Options::=--force-confdef', '--allow-unauthenticated',
'--assume-yes', 'install', 'unzip']' returned 100. Reading package
lists...
Building dependency tree...
Reading state information...
Suggested packages:
zip
The following NEW packages will be installed:
unzip
0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
Need to get 157 kB of archives.
After this operation, 391 kB of additional disk space will be used.
Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main unzip amd64
6.0-9ubuntu1.3
Could not resolve 'archive.ubuntu.com'
Err http://security.ubuntu.com/ubuntu/ trusty-security/main unzip
amd64 6.0-9ubuntu1.3
Could not resolve 'security.ubuntu.com'
E: Failed to fetch
http://security.ubuntu.com/ubuntu/pool/main/u/unzip/unzip_6.0-9ubuntu1.3_amd64.deb
Could not resolve 'security.ubuntu.com'
E: Unable to fetch some archives, maybe run apt-get update or try with
--fix-missing?
2015-08-20 06:53:13,514 - Failed to install package unzip. Executing
`/usr/bin/apt-get update -qq`
2015-08-20 06:53:14,780 - Retrying to install package unzip
Re: Issue with deploying Cluster through Ambari Agent behind proxy
Posted by Dmitry Sen <ds...@hortonworks.com>.
Hi Konrad,
There are a lot o ways to setup user environment in Linux. Please create new issue at https://issues.apache.org if you think, there is a bug in ambari agent, also add detailed steps how you setup environmental variables, ambari and OS versions.
You can not rely on environmental variables, but edit apt.conf configuration file.
echo 'Acquire::http::Proxy "http://<proxy_host>:<proxy_port>/";' >> /etc/apt/apt.conf
?
BR,
Dmytro Sen
?
________________________________
From: Konrad Hübner <ko...@gmail.com>
Sent: Friday, August 21, 2015 10:32 AM
To: Dmitry Sen
Cc: user@ambari.apache.org
Subject: Re: Issue with deploying Cluster through Ambari Agent behind proxy
Hi,
setting up a proxy on all agents had been done before and did not help. What I finally figured out was that by manipulating the ambari-sudo.sh script on each agent to export the proxy environment variables before actually executing the commands helped. I have not gone into further analysis, but it seems that the script kills the users environment variables, at least those on /etc/environment. Maybe that is worth a ticket. Will see if I can figure out in more detail what's going wrong.
Best
-Konrad
2015-08-20 9:44 GMT+02:00 Dmitry Sen <ds...@hortonworks.com>>:
Hi,
There are 2 options
1. Setup apt to use proxy on all agents (not sure if it works is your local DNS doesn't resolve public hostnames)
echo 'Acquire::http::Proxy "http://<proxy_host>:<proxy_port>/";' >> /etc/apt/apt.conf
2. Create a local copy of repositories, required for installing Hadoop Stack
http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_Installing_HDP_AMB/content/_using_a_local_repository.html
________________________________
From: Konrad Hübner <ko...@gmail.com>>
Sent: Thursday, August 20, 2015 10:20 AM
To: user@ambari.apache.org<ma...@ambari.apache.org>
Subject: Fwd: Issue with deploying Cluster through Ambari Agent behind proxy
Hi,
I try to deploy a HDP 2.3 Cluster with Ambari 2.1. By force I am behind a proxy server. The proxy is configured both systemwide in /etc/env, including export of environment variables for sudoers and in /etc/apt/apt.conf.d with Accquire::proxy command. Ambari Server was configured to use a non-root users with sudo privilege. Installing ambari client worked, but when starting the deployment of the cluster through the agent, the agent runs apt-get install and that failes when resolving the hostname. This looks like it is not using the proxy. Please find the log output below. Any help is appreciated!
Thanks
-Konrad
2015-08-20 06:53:13,299 - Package['unzip'] {}
2015-08-20 06:53:13,310 - Installing package unzip ('/usr/bin/apt-get -q -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install unzip')
2015-08-20 06:53:13,514 - Execution of '['/usr/bin/apt-get', '-q', '-o', 'Dpkg::Options::=--force-confdef', '--allow-unauthenticated', '--assume-yes', 'install', 'unzip']' returned 100. Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
zip
The following NEW packages will be installed:
unzip
0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
Need to get 157 kB of archives.
After this operation, 391 kB of additional disk space will be used.
Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main unzip amd64 6.0-9ubuntu1.3
Could not resolve 'archive.ubuntu.com<http://archive.ubuntu.com>'
Err http://security.ubuntu.com/ubuntu/ trusty-security/main unzip amd64 6.0-9ubuntu1.3
Could not resolve 'security.ubuntu.com<http://security.ubuntu.com>'
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/u/unzip/unzip_6.0-9ubuntu1.3_amd64.deb Could not resolve 'security.ubuntu.com<http://security.ubuntu.com>'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
2015-08-20 06:53:13,514 - Failed to install package unzip. Executing `/usr/bin/apt-get update -qq`
2015-08-20 06:53:14,780 - Retrying to install package unzip
Re: Issue with deploying Cluster through Ambari Agent behind proxy
Posted by Konrad Hübner <ko...@gmail.com>.
Hi,
setting up a proxy on all agents had been done before and did not help.
What I finally figured out was that by manipulating the ambari-sudo.sh
script on each agent to export the proxy environment variables before
actually executing the commands helped. I have not gone into further
analysis, but it seems that the script kills the users environment
variables, at least those on /etc/environment. Maybe that is worth a
ticket. Will see if I can figure out in more detail what's going wrong.
Best
-Konrad
2015-08-20 9:44 GMT+02:00 Dmitry Sen <ds...@hortonworks.com>:
> Hi,
>
>
> There are 2 options
>
> 1. Setup apt to use proxy on all agents (not sure if it works is your
> local DNS doesn't resolve public hostnames)
>
> echo 'Acquire::http::Proxy "http://<proxy_host>:<proxy_port>/";' >>
> /etc/apt/apt.conf
>
> 2. Create a local copy of repositories, required for installing Hadoop
> Stack
>
>
> http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_Installing_HDP_AMB/content/_using_a_local_repository.html
>
>
>
> ------------------------------
> *From:* Konrad Hübner <ko...@gmail.com>
> *Sent:* Thursday, August 20, 2015 10:20 AM
> *To:* user@ambari.apache.org
> *Subject:* Fwd: Issue with deploying Cluster through Ambari Agent behind
> proxy
>
> Hi,
>
> I try to deploy a HDP 2.3 Cluster with Ambari 2.1. By force I am behind a
> proxy server. The proxy is configured both systemwide in /etc/env,
> including export of environment variables for sudoers and in
> /etc/apt/apt.conf.d with Accquire::proxy command. Ambari Server was
> configured to use a non-root users with sudo privilege. Installing ambari
> client worked, but when starting the deployment of the cluster through the
> agent, the agent runs apt-get install and that failes when resolving the
> hostname. This looks like it is not using the proxy. Please find the log
> output below. Any help is appreciated!
>
> Thanks
> -Konrad
>
> 2015-08-20 06:53:13,299 - Package['unzip'] {}
> 2015-08-20 06:53:13,310 - Installing package unzip ('/usr/bin/apt-get -q -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install unzip')
> 2015-08-20 06:53:13,514 - Execution of '['/usr/bin/apt-get', '-q', '-o', 'Dpkg::Options::=--force-confdef', '--allow-unauthenticated', '--assume-yes', 'install', 'unzip']' returned 100. Reading package lists...
> Building dependency tree...
> Reading state information...
> Suggested packages:
> zip
> The following NEW packages will be installed:
> unzip
> 0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
> Need to get 157 kB of archives.
> After this operation, 391 kB of additional disk space will be used.
> Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main unzip amd64 6.0-9ubuntu1.3
> Could not resolve 'archive.ubuntu.com'
> Err http://security.ubuntu.com/ubuntu/ trusty-security/main unzip amd64 6.0-9ubuntu1.3
> Could not resolve 'security.ubuntu.com'
> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/u/unzip/unzip_6.0-9ubuntu1.3_amd64.deb Could not resolve 'security.ubuntu.com'
>
> E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
> 2015-08-20 06:53:13,514 - Failed to install package unzip. Executing `/usr/bin/apt-get update -qq`
> 2015-08-20 06:53:14,780 - Retrying to install package unzip
>
>
>
>
Re: Issue with deploying Cluster through Ambari Agent behind proxy
Posted by Dmitry Sen <ds...@hortonworks.com>.
Hi,
There are 2 options
1. Setup apt to use proxy on all agents (not sure if it works is your local DNS doesn't resolve public hostnames)
echo 'Acquire::http::Proxy "http://<proxy_host>:<proxy_port>/";' >> /etc/apt/apt.conf
2. Create a local copy of repositories, required for installing Hadoop Stack
http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_Installing_HDP_AMB/content/_using_a_local_repository.html
________________________________
From: Konrad Hübner <ko...@gmail.com>
Sent: Thursday, August 20, 2015 10:20 AM
To: user@ambari.apache.org
Subject: Fwd: Issue with deploying Cluster through Ambari Agent behind proxy
Hi,
I try to deploy a HDP 2.3 Cluster with Ambari 2.1. By force I am behind a proxy server. The proxy is configured both systemwide in /etc/env, including export of environment variables for sudoers and in /etc/apt/apt.conf.d with Accquire::proxy command. Ambari Server was configured to use a non-root users with sudo privilege. Installing ambari client worked, but when starting the deployment of the cluster through the agent, the agent runs apt-get install and that failes when resolving the hostname. This looks like it is not using the proxy. Please find the log output below. Any help is appreciated!
Thanks
-Konrad
2015-08-20 06:53:13,299 - Package['unzip'] {}
2015-08-20 06:53:13,310 - Installing package unzip ('/usr/bin/apt-get -q -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install unzip')
2015-08-20 06:53:13,514 - Execution of '['/usr/bin/apt-get', '-q', '-o', 'Dpkg::Options::=--force-confdef', '--allow-unauthenticated', '--assume-yes', 'install', 'unzip']' returned 100. Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
zip
The following NEW packages will be installed:
unzip
0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
Need to get 157 kB of archives.
After this operation, 391 kB of additional disk space will be used.
Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main unzip amd64 6.0-9ubuntu1.3
Could not resolve 'archive.ubuntu.com<http://archive.ubuntu.com>'
Err http://security.ubuntu.com/ubuntu/ trusty-security/main unzip amd64 6.0-9ubuntu1.3
Could not resolve 'security.ubuntu.com<http://security.ubuntu.com>'
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/u/unzip/unzip_6.0-9ubuntu1.3_amd64.deb Could not resolve 'security.ubuntu.com<http://security.ubuntu.com>'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
2015-08-20 06:53:13,514 - Failed to install package unzip. Executing `/usr/bin/apt-get update -qq`
2015-08-20 06:53:14,780 - Retrying to install package unzip