You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Karam Singh (JIRA)" <ji...@apache.org> on 2008/05/08 14:14:55 UTC
[jira] Created: (HADOOP-3367) hod dealliocate does execute properly
and throws Operation not permitted error, when hod allcoated cluster is
shared with multiple users.
hod dealliocate does execute properly and throws Operation not permitted error, when hod allcoated cluster is shared with multiple users.
-----------------------------------------------------------------------------------------------------------------------------------------
Key: HADOOP-3367
URL: https://issues.apache.org/jira/browse/HADOOP-3367
Project: Hadoop Core
Issue Type: Bug
Components: contrib/hod
Affects Versions: 0.17.0
Reporter: Karam Singh
Priority: Minor
1. Allocate cluster using hadoop with dfs permissions on and this cluster is used by two users.
2. Ran randomtextwriter and distcp jobs.
3. When tried to deallocate hod deallocate threw "Operation not permitted" but exitted with exit code 0.
Following the output of deallocate operation -:
[
[2008-05-07 15:01:47,503] DEBUG/10 hadoop:595 - hadoop-ui-log-dir not specified. Skipping Hadoop UI log collection.
[2008-05-07 15:01:47,512] DEBUG/10 hadoop:616 - calling rm.stop
[2008-05-07 15:01:47,559] DEBUG/10 hadoop:618 - completed rm.stop
[2008-05-07 15:01:47,564] CRITICAL/50 hod:517 - op: deallocate cluster_dir failed: <type 'exceptions.OSError'> [Errno 1] Operation not permitted: '<path of hod.temp-dir>/<userid>.<cluster_id>'
[2008-05-07 15:01:47,569] DEBUG/10 hod:518 - Traceback (most recent call last):
File "/grid/0/hodqa/hod/hod-dev-20080414/hodlib/Hod/hod.py", line 510, in operation
getattr(self, "_op_%s" % opList[0])(opList)
File "/grid/0/hodqa/hod/hod-dev-20080414/hodlib/Hod/hod.py", line 365, in _op_deallocate
self.__cluster.deallocate(clusterDir, clusterInfo)
File "/grid/0/hodqa/hod/hod-dev-20080414/hodlib/Hod/hadoop.py", line 624, in deallocate
shutil.rmtree(tempDir)
File "/export/crawlspace/kryptonite/comps//python-2.5.1/lib/python2.5/shutil.py", line 178, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/export/crawlspace/kryptonite/comps//python-2.5.1/lib/python2.5/shutil.py", line 176, in rmtree
os.rmdir(path)
OSError: [Errno 1] Operation not permitted: '<path of hod.temp-dir>/<userid>.<clusrter_id>'
[2008-05-07 15:01:47,511] DEBUG/10 hod:522 - return code: 0
]
Torque got comleted, hod list shows clsuter as dead cluster.
It seems when mapred job is run by other user then the user who allocated the cluster. hdo.temp-dir is getting created with ownership of mapred who ran maped jobs.
So when deallocate operation is fired, by trhe user who allcoated the cluser, hod tries to removes <hod.temp-dir>/<useruid>.<cluster_id> durectory which fails causing dellocate operation to behave oddly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3367) hod dealliocate does not xecute
properly and throws Operation not permitted error, when hod allcoated
cluster is shared with multiple users.
Posted by "Karam Singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karam Singh updated HADOOP-3367:
--------------------------------
Summary: hod dealliocate does not xecute properly and throws Operation not permitted error, when hod allcoated cluster is shared with multiple users. (was: hod dealliocate does execute properly and throws Operation not permitted error, when hod allcoated cluster is shared with multiple users.)
> hod dealliocate does not xecute properly and throws Operation not permitted error, when hod allcoated cluster is shared with multiple users.
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3367
> URL: https://issues.apache.org/jira/browse/HADOOP-3367
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.17.0
> Reporter: Karam Singh
> Priority: Minor
>
> 1. Allocate cluster using hadoop with dfs permissions on and this cluster is used by two users.
> 2. Ran randomtextwriter and distcp jobs.
> 3. When tried to deallocate hod deallocate threw "Operation not permitted" but exitted with exit code 0.
> Following the output of deallocate operation -:
> [
> [2008-05-07 15:01:47,503] DEBUG/10 hadoop:595 - hadoop-ui-log-dir not specified. Skipping Hadoop UI log collection.
> [2008-05-07 15:01:47,512] DEBUG/10 hadoop:616 - calling rm.stop
> [2008-05-07 15:01:47,559] DEBUG/10 hadoop:618 - completed rm.stop
> [2008-05-07 15:01:47,564] CRITICAL/50 hod:517 - op: deallocate cluster_dir failed: <type 'exceptions.OSError'> [Errno 1] Operation not permitted: '<path of hod.temp-dir>/<userid>.<cluster_id>'
> [2008-05-07 15:01:47,569] DEBUG/10 hod:518 - Traceback (most recent call last):
> File "/grid/0/hodqa/hod/hod-dev-20080414/hodlib/Hod/hod.py", line 510, in operation
> getattr(self, "_op_%s" % opList[0])(opList)
> File "/grid/0/hodqa/hod/hod-dev-20080414/hodlib/Hod/hod.py", line 365, in _op_deallocate
> self.__cluster.deallocate(clusterDir, clusterInfo)
> File "/grid/0/hodqa/hod/hod-dev-20080414/hodlib/Hod/hadoop.py", line 624, in deallocate
> shutil.rmtree(tempDir)
> File "/export/crawlspace/kryptonite/comps//python-2.5.1/lib/python2.5/shutil.py", line 178, in rmtree
> onerror(os.rmdir, path, sys.exc_info())
> File "/export/crawlspace/kryptonite/comps//python-2.5.1/lib/python2.5/shutil.py", line 176, in rmtree
> os.rmdir(path)
> OSError: [Errno 1] Operation not permitted: '<path of hod.temp-dir>/<userid>.<clusrter_id>'
> [2008-05-07 15:01:47,511] DEBUG/10 hod:522 - return code: 0
> ]
> Torque got comleted, hod list shows clsuter as dead cluster.
> It seems when mapred job is run by other user then the user who allocated the cluster. hdo.temp-dir is getting created with ownership of mapred who ran maped jobs.
> So when deallocate operation is fired, by trhe user who allcoated the cluser, hod tries to removes <hod.temp-dir>/<useruid>.<cluster_id> durectory which fails causing dellocate operation to behave oddly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.