You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Pawel Akonom <pa...@vnomic.com> on 2016/06/15 19:23:25 UTC
Fw: HDP install issues about hdp-select
Hi,
I have some problems with hadoop cluster scratch installation (not upgrade). Versions I am using:
Ambari version: 2.2.2.0
HDP stack version: 2.3.4.7
Zookeper installation fails on step:
Execute['ambari-sudo.sh /usr/bin/hdp-select set all `ambari-python-wrap /usr/bin/hdp-select versions | grep ^2.3 | tail -1`'] {'only_if': 'ls -d /usr/hdp/2.3*'}
When I execute this command manually on bash I get error:
[root@hdp-vora-master ~]# ambari-python-wrap /usr/bin/hdp-select versions
Traceback (most recent call last):
File "/usr/bin/hdp-select", line 378, in <module>
printVersions()
File "/usr/bin/hdp-select", line 235, in printVersions
result[tuple(map(int, versionRegex.split(f)))] = f
ValueError: invalid literal for int() with base 10: 'hadoop'
The same problem is described in Hortonworks community:
https://community.hortonworks.com/questions/5811/install-of-hdp-fails-with-valueerror-invalid-liter.html
Workaround for this problem is to edit /usr/bin/hdp-select python script and modify printVersions() function.
Function before:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
result[tuple(map(int, versionRegex.split(f)))] = f
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Function after modification:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
try:
result[tuple(map(int, versionRegex.split(f)))] = f
except:
pass
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Hadoop cluster installation need to be automated so this workaround is not a solution. Script /usr/bin/hdp-select appears during installation and it doesn't come with any rpm package. It can be a bug in the script or maybe it fails only with some specific python versions only.
Do you know the problem? If so what can be the solution?
Thanks in advance,
Pawel
Re: HDP install issues about hdp-select
Posted by Pawel Akonom <pa...@vnomic.com>.
Hi Jonathan,
I have change "ZooKeeper directory" directory setting on step 7 "Customize services" on ambari console. By default it is set to "/usr/hdp/hadoop/zookeeper" and I have changed it to "/var/lib/hadoop/zookeeper". After that I didn't get error from /usr/bin/hdp-select script.
In my opinion it is a bug and it could be fixed in two ways:
1. Change default "ZooKeeper directory" to somewhere else than "/usr/hdp/hadoop/zookeeper"
or
2. /usr/bin/hdp-select should expect in "/usr/hdp" directories like "current", "hadoop" and version dir.
Thank you for helpful hint.
Br,
Pawel
________________________________
From: Pawel Akonom <pa...@vnomic.com>
Sent: Thursday, June 16, 2016 5:25:07 PM
To: user@ambari.apache.org
Cc: Derek Palma
Subject: Re: HDP install issues about hdp-select
Hi Jonathan,
Thank you for your response. I have checked /usr/hdp directory on all the hadoop cluster nodes. Before installation it was empty and when it crashed I have checked the directory on all the nodes. On one node there is additional directory created:
[root@hdp-vora-master hdp]# ls -la /usr/hdp/
total 20
drwxr-xr-x. 5 root root 4096 Jun 16 10:20 .
drwxr-xr-x. 15 root root 4096 Jun 16 10:18 ..
drwxr-xr-x. 11 root root 4096 Jun 16 10:20 2.3.4.7-4
drwxr-xr-x. 2 root root 4096 Jun 16 10:20 current
drwxr-xr-x. 3 root root 4096 Jun 16 10:20 hadoop
there is one directory and one file in /usr/hdp/hadoop:
[root@hdp-vora-master hdp]# ls -la /usr/hdp/hadoop/zookeeper/myid
-rw-r--r--. 1 root root 1 Jun 16 10:20 /usr/hdp/hadoop/zookeeper/myid
[root@hdp-vora-master hdp]# cat /usr/hdp/hadoop/zookeeper/myid
2
It looks like zookeeper have created it. I didn't change anything in default zookeeper configuration during installation. Maybe I should change something to create myid file in different directory.
Br,
Pawel
________________________________
From: Jonathan Hurley <jh...@hortonworks.com>
Sent: Wednesday, June 15, 2016 11:03 PM
To: user@ambari.apache.org
Cc: Derek Palma
Subject: Re: HDP install issues about hdp-select
I believe this is because you have a "hadoop" directory in /usr/hdp ... /usr/hdp should only contain versions and "current". If there's another directory, it would cause the hdp-select tool to fail.
On Jun 15, 2016, at 3:23 PM, Pawel Akonom <pa...@vnomic.com>> wrote:
Hi,
I have some problems with hadoop cluster scratch installation (not upgrade). Versions I am using:
Ambari version: 2.2.2.0
HDP stack version: 2.3.4.7
Zookeper installation fails on step:
Execute['ambari-sudo.sh /usr/bin/hdp-select set all `ambari-python-wrap /usr/bin/hdp-select versions | grep ^2.3 | tail -1`'] {'only_if': 'ls -d /usr/hdp/2.3*'}
When I execute this command manually on bash I get error:
[root@hdp-vora-master ~]# ambari-python-wrap /usr/bin/hdp-select versions
Traceback (most recent call last):
File "/usr/bin/hdp-select", line 378, in <module>
printVersions()
File "/usr/bin/hdp-select", line 235, in printVersions
result[tuple(map(int, versionRegex.split(f)))] = f
ValueError: invalid literal for int() with base 10: 'hadoop'
The same problem is described in Hortonworks community:
https://community.hortonworks.com/questions/5811/install-of-hdp-fails-with-valueerror-invalid-liter.html
Workaround for this problem is to edit /usr/bin/hdp-select python script and modify printVersions() function.
Function before:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
result[tuple(map(int, versionRegex.split(f)))] = f
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Function after modification:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
try:
result[tuple(map(int, versionRegex.split(f)))] = f
except:
pass
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Hadoop cluster installation need to be automated so this workaround is not a solution. Script /usr/bin/hdp-select appears during installation and it doesn't come with any rpm package. It can be a bug in the script or maybe it fails only with some specific python versions only.
Do you know the problem? If so what can be the solution?
Thanks in advance,
Pawel
Re: HDP install issues about hdp-select
Posted by Jonathan Hurley <jh...@hortonworks.com>.
A default HDP installation of ZooKeeper specifies the data directory as "/hadoop/zookeeper". You have it as /usr/hdp/hadoop/zookeeper. You can change this in the zoo.cfg/dataDir property. I believe Ambari exposes this as "ZooKeeper directory" property in the UI.
Taking a look at the stack for HDP 2.3.4.7, we can see this in the zoo.cfg file:
<property>
<name>dataDir</name>
<value>/hadoop/zookeeper</value>
<display-name>ZooKeeper directory</display-name>
<description>Data directory for ZooKeeper.</description>
So something must be customizing it in your deployment.
On Jun 16, 2016, at 10:25 AM, Pawel Akonom <pa...@vnomic.com>> wrote:
Hi Jonathan,
Thank you for your response. I have checked /usr/hdp directory on all the hadoop cluster nodes. Before installation it was empty and when it crashed I have checked the directory on all the nodes. On one node there is additional directory created:
[root@hdp-vora-master hdp]# ls -la /usr/hdp/
total 20
drwxr-xr-x. 5 root root 4096 Jun 16 10:20 .
drwxr-xr-x. 15 root root 4096 Jun 16 10:18 ..
drwxr-xr-x. 11 root root 4096 Jun 16 10:20 2.3.4.7-4
drwxr-xr-x. 2 root root 4096 Jun 16 10:20 current
drwxr-xr-x. 3 root root 4096 Jun 16 10:20 hadoop
there is one directory and one file in /usr/hdp/hadoop:
[root@hdp-vora-master hdp]# ls -la /usr/hdp/hadoop/zookeeper/myid
-rw-r--r--. 1 root root 1 Jun 16 10:20 /usr/hdp/hadoop/zookeeper/myid
[root@hdp-vora-master hdp]# cat /usr/hdp/hadoop/zookeeper/myid
2
It looks like zookeeper have created it. I didn't change anything in default zookeeper configuration during installation. Maybe I should change something to create myid file in different directory.
Br,
Pawel
________________________________
From: Jonathan Hurley <jh...@hortonworks.com>>
Sent: Wednesday, June 15, 2016 11:03 PM
To: user@ambari.apache.org<ma...@ambari.apache.org>
Cc: Derek Palma
Subject: Re: HDP install issues about hdp-select
I believe this is because you have a "hadoop" directory in /usr/hdp ... /usr/hdp should only contain versions and "current". If there's another directory, it would cause the hdp-select tool to fail.
On Jun 15, 2016, at 3:23 PM, Pawel Akonom <pa...@vnomic.com>> wrote:
Hi,
I have some problems with hadoop cluster scratch installation (not upgrade). Versions I am using:
Ambari version: 2.2.2.0
HDP stack version: 2.3.4.7
Zookeper installation fails on step:
Execute['ambari-sudo.sh /usr/bin/hdp-select set all `ambari-python-wrap /usr/bin/hdp-select versions | grep ^2.3 | tail -1`'] {'only_if': 'ls -d /usr/hdp/2.3*'}
When I execute this command manually on bash I get error:
[root@hdp-vora-master ~]# ambari-python-wrap /usr/bin/hdp-select versions
Traceback (most recent call last):
File "/usr/bin/hdp-select", line 378, in <module>
printVersions()
File "/usr/bin/hdp-select", line 235, in printVersions
result[tuple(map(int, versionRegex.split(f)))] = f
ValueError: invalid literal for int() with base 10: 'hadoop'
The same problem is described in Hortonworks community:
https://community.hortonworks.com/questions/5811/install-of-hdp-fails-with-valueerror-invalid-liter.html
Workaround for this problem is to edit /usr/bin/hdp-select python script and modify printVersions() function.
Function before:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
result[tuple(map(int, versionRegex.split(f)))] = f
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Function after modification:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
try:
result[tuple(map(int, versionRegex.split(f)))] = f
except:
pass
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Hadoop cluster installation need to be automated so this workaround is not a solution. Script /usr/bin/hdp-select appears during installation and it doesn't come with any rpm package. It can be a bug in the script or maybe it fails only with some specific python versions only.
Do you know the problem? If so what can be the solution?
Thanks in advance,
Pawel
Re: HDP install issues about hdp-select
Posted by Pawel Akonom <pa...@vnomic.com>.
Hi Jonathan,
Thank you for your response. I have checked /usr/hdp directory on all the hadoop cluster nodes. Before installation it was empty and when it crashed I have checked the directory on all the nodes. On one node there is additional directory created:
[root@hdp-vora-master hdp]# ls -la /usr/hdp/
total 20
drwxr-xr-x. 5 root root 4096 Jun 16 10:20 .
drwxr-xr-x. 15 root root 4096 Jun 16 10:18 ..
drwxr-xr-x. 11 root root 4096 Jun 16 10:20 2.3.4.7-4
drwxr-xr-x. 2 root root 4096 Jun 16 10:20 current
drwxr-xr-x. 3 root root 4096 Jun 16 10:20 hadoop
there is one directory and one file in /usr/hdp/hadoop:
[root@hdp-vora-master hdp]# ls -la /usr/hdp/hadoop/zookeeper/myid
-rw-r--r--. 1 root root 1 Jun 16 10:20 /usr/hdp/hadoop/zookeeper/myid
[root@hdp-vora-master hdp]# cat /usr/hdp/hadoop/zookeeper/myid
2
It looks like zookeeper have created it. I didn't change anything in default zookeeper configuration during installation. Maybe I should change something to create myid file in different directory.
Br,
Pawel
________________________________
From: Jonathan Hurley <jh...@hortonworks.com>
Sent: Wednesday, June 15, 2016 11:03 PM
To: user@ambari.apache.org
Cc: Derek Palma
Subject: Re: HDP install issues about hdp-select
I believe this is because you have a "hadoop" directory in /usr/hdp ... /usr/hdp should only contain versions and "current". If there's another directory, it would cause the hdp-select tool to fail.
On Jun 15, 2016, at 3:23 PM, Pawel Akonom <pa...@vnomic.com>> wrote:
Hi,
I have some problems with hadoop cluster scratch installation (not upgrade). Versions I am using:
Ambari version: 2.2.2.0
HDP stack version: 2.3.4.7
Zookeper installation fails on step:
Execute['ambari-sudo.sh /usr/bin/hdp-select set all `ambari-python-wrap /usr/bin/hdp-select versions | grep ^2.3 | tail -1`'] {'only_if': 'ls -d /usr/hdp/2.3*'}
When I execute this command manually on bash I get error:
[root@hdp-vora-master ~]# ambari-python-wrap /usr/bin/hdp-select versions
Traceback (most recent call last):
File "/usr/bin/hdp-select", line 378, in <module>
printVersions()
File "/usr/bin/hdp-select", line 235, in printVersions
result[tuple(map(int, versionRegex.split(f)))] = f
ValueError: invalid literal for int() with base 10: 'hadoop'
The same problem is described in Hortonworks community:
https://community.hortonworks.com/questions/5811/install-of-hdp-fails-with-valueerror-invalid-liter.html
Workaround for this problem is to edit /usr/bin/hdp-select python script and modify printVersions() function.
Function before:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
result[tuple(map(int, versionRegex.split(f)))] = f
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Function after modification:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
try:
result[tuple(map(int, versionRegex.split(f)))] = f
except:
pass
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Hadoop cluster installation need to be automated so this workaround is not a solution. Script /usr/bin/hdp-select appears during installation and it doesn't come with any rpm package. It can be a bug in the script or maybe it fails only with some specific python versions only.
Do you know the problem? If so what can be the solution?
Thanks in advance,
Pawel
Re: HDP install issues about hdp-select
Posted by Jonathan Hurley <jh...@hortonworks.com>.
I believe this is because you have a "hadoop" directory in /usr/hdp ... /usr/hdp should only contain versions and "current". If there's another directory, it would cause the hdp-select tool to fail.
On Jun 15, 2016, at 3:23 PM, Pawel Akonom <pa...@vnomic.com>> wrote:
Hi,
I have some problems with hadoop cluster scratch installation (not upgrade). Versions I am using:
Ambari version: 2.2.2.0
HDP stack version: 2.3.4.7
Zookeper installation fails on step:
Execute['ambari-sudo.sh /usr/bin/hdp-select set all `ambari-python-wrap /usr/bin/hdp-select versions | grep ^2.3 | tail -1`'] {'only_if': 'ls -d /usr/hdp/2.3*'}
When I execute this command manually on bash I get error:
[root@hdp-vora-master ~]# ambari-python-wrap /usr/bin/hdp-select versions
Traceback (most recent call last):
File "/usr/bin/hdp-select", line 378, in <module>
printVersions()
File "/usr/bin/hdp-select", line 235, in printVersions
result[tuple(map(int, versionRegex.split(f)))] = f
ValueError: invalid literal for int() with base 10: 'hadoop'
The same problem is described in Hortonworks community:
https://community.hortonworks.com/questions/5811/install-of-hdp-fails-with-valueerror-invalid-liter.html
Workaround for this problem is to edit /usr/bin/hdp-select python script and modify printVersions() function.
Function before:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
result[tuple(map(int, versionRegex.split(f)))] = f
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Function after modification:
# Print the installed packages
def printVersions():
result = {}
for f in os.listdir(root):
if f not in [".", "..", "current", "share", "lost+found"]:
try:
result[tuple(map(int, versionRegex.split(f)))] = f
except:
pass
keys = result.keys()
keys.sort()
for k in keys:
print result[k]
Hadoop cluster installation need to be automated so this workaround is not a solution. Script /usr/bin/hdp-select appears during installation and it doesn't come with any rpm package. It can be a bug in the script or maybe it fails only with some specific python versions only.
Do you know the problem? If so what can be the solution?
Thanks in advance,
Pawel