You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ravi Phulari (JIRA)" <ji...@apache.org> on 2010/04/13 23:10:01 UTC
[jira] Created: (HADOOP-6702) Incorrect exit codes for "dfs
-chown", "dfs -chgrp" when input is given in wildcard format.
Incorrect exit codes for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
--------------------------------------------------------------------------------------------
Key: HADOOP-6702
URL: https://issues.apache.org/jira/browse/HADOOP-6702
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 0.20.2, 0.20.1, 0.20.0, 0.19.1
Reporter: Ravi Phulari
Assignee: Ravi Phulari
Priority: Minor
Fix For: 0.20.3, 0.21.0, 0.22.0
Currently incorrect exit codes are given for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
This bug is due to missing update of errors count in {{FsShell.java}}.
{code:title=FsShell.java|borderStyle=solid}
int runCmdHandler(CmdHandler handler, String[] args,
int startIndex, boolean recursive)
throws IOException {
int errors = 0;
for (int i=startIndex; i<args.length; i++) {
Path srcPath = new Path(args[i]);
FileSystem srcFs = srcPath.getFileSystem(getConf());
Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);
for(Path path : paths) {
try {
FileStatus file = srcFs.getFileStatus(path);
if (file == null) {
System.err.println(handler.getName() +
": could not get status for '" + path + "'");
errors++;
} else {
errors += runCmdHandler(handler, file, srcFs, recursive);
}
} catch (IOException e) {
String msg = (e.getMessage() != null ? e.getLocalizedMessage() :
(e.getCause().getMessage() != null ?
e.getCause().getLocalizedMessage() : "null"));
System.err.println(handler.getName() + ": could not get status for '"
+ path + "': " + msg.split("\n")[0]);
errors++;
}
}
}
{code}
If there are no files on HDFS matching to wildcard input then {{srcFs.globStatus(srcpath)}} returns 0.
{{ Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);}}
Resulting no increment in {{errors}} and command exits with 0 even though file/directory does not exist.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-6702) Incorrect exit codes for "dfs
-chown", "dfs -chgrp" when input is given in wildcard format.
Posted by "Ravi Phulari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Phulari resolved HADOOP-6702.
----------------------------------
Resolution: Fixed
HADOOP-6701 Fixes this issue.
> Incorrect exit codes for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-6702
> URL: https://issues.apache.org/jira/browse/HADOOP-6702
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
> Reporter: Ravi Phulari
> Assignee: Ravi Phulari
> Priority: Minor
> Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> Currently incorrect exit codes are given for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
> This bug is due to missing update of errors count in {{FsShell.java}}.
> {code:title=FsShell.java|borderStyle=solid}
> int runCmdHandler(CmdHandler handler, String[] args,
> int startIndex, boolean recursive)
> throws IOException {
> int errors = 0;
>
> for (int i=startIndex; i<args.length; i++) {
> Path srcPath = new Path(args[i]);
> FileSystem srcFs = srcPath.getFileSystem(getConf());
> Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);
> for(Path path : paths) {
> try {
> FileStatus file = srcFs.getFileStatus(path);
> if (file == null) {
> System.err.println(handler.getName() +
> ": could not get status for '" + path + "'");
> errors++;
> } else {
> errors += runCmdHandler(handler, file, srcFs, recursive);
> }
> } catch (IOException e) {
> String msg = (e.getMessage() != null ? e.getLocalizedMessage() :
> (e.getCause().getMessage() != null ?
> e.getCause().getLocalizedMessage() : "null"));
> System.err.println(handler.getName() + ": could not get status for '"
> + path + "': " + msg.split("\n")[0]);
> errors++;
> }
> }
> }
> {code}
> If there are no files on HDFS matching to wildcard input then {{srcFs.globStatus(srcpath)}} returns 0.
> {{ Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);}}
> Resulting no increment in {{errors}} and command exits with 0 even though file/directory does not exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6702) Incorrect exit codes for "dfs
-chown", "dfs -chgrp" when input is given in wildcard format.
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857862#action_12857862 ]
Koji Noguchi commented on HADOOP-6702:
--------------------------------------
Incorrect return code for wildcard in hadoop is not limited to chown/chgrp.
It's everywhere.
For example,
In 'ls', this is how unix performs,
{noformat}
% ls nonexist*
ls: No match.
% echo $?
1
% ls nonexist* file*
fileA
% echo $?
0
% ls file* nonexist*
fileA
% echo $?
0
{noformat}
*It returns 0 as long as one of the globing matches*.
and in hadoop 'ls'
{noformat}
% hadoop dfs -ls file\* nonexist\*
Found 1 items
-rw------- 3 knoguchi users 7 2010-04-08 15:57 /user/knoguchi/fileA
ls: Cannot access nonexist*: No such file or directory.
% echo $?
255
% hadoop dfs -ls nonexist\* file\*
ls: Cannot access nonexist*: No such file or directory.
Found 1 items
-rw------- 3 knoguchi users 7 2010-04-08 15:57 /user/knoguchi/fileA
% echo $?
0
%
{noformat}
hadoop 'ls' simply returns the result of the last globbing.
This behavior is also inconsistent from command to command.
Picking three hadoop commands. chgrp/ls/du.
|| command || single globbing || multiple globbing ||
| chown/chgrp/etc | X (returns 0 even if globing returns empty) | X (returns 0 even if all the globbing returns empty) |
| ls | O | X (returns last globbing result) |
| du | O | X (returns non-zero even if one of the globbing fail) |
Suggested fix in this Jira would simply change the behavior of 'chown/chgrp' to 'du'-like which means multiple globbing will still be incorrect.
> Incorrect exit codes for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-6702
> URL: https://issues.apache.org/jira/browse/HADOOP-6702
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
> Reporter: Ravi Phulari
> Assignee: Ravi Phulari
> Priority: Minor
> Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> Currently incorrect exit codes are given for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
> This bug is due to missing update of errors count in {{FsShell.java}}.
> {code:title=FsShell.java|borderStyle=solid}
> int runCmdHandler(CmdHandler handler, String[] args,
> int startIndex, boolean recursive)
> throws IOException {
> int errors = 0;
>
> for (int i=startIndex; i<args.length; i++) {
> Path srcPath = new Path(args[i]);
> FileSystem srcFs = srcPath.getFileSystem(getConf());
> Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);
> for(Path path : paths) {
> try {
> FileStatus file = srcFs.getFileStatus(path);
> if (file == null) {
> System.err.println(handler.getName() +
> ": could not get status for '" + path + "'");
> errors++;
> } else {
> errors += runCmdHandler(handler, file, srcFs, recursive);
> }
> } catch (IOException e) {
> String msg = (e.getMessage() != null ? e.getLocalizedMessage() :
> (e.getCause().getMessage() != null ?
> e.getCause().getLocalizedMessage() : "null"));
> System.err.println(handler.getName() + ": could not get status for '"
> + path + "': " + msg.split("\n")[0]);
> errors++;
> }
> }
> }
> {code}
> If there are no files on HDFS matching to wildcard input then {{srcFs.globStatus(srcpath)}} returns 0.
> {{ Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);}}
> Resulting no increment in {{errors}} and command exits with 0 even though file/directory does not exist.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-6702) Incorrect exit codes for "dfs
-chown", "dfs -chgrp" when input is given in wildcard format.
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865958#action_12865958 ]
Koji Noguchi commented on HADOOP-6702:
--------------------------------------
bq. HADOOP-6701 Fixes this issue.
Ravi, HADOOP-6701 just moved the error handling of globbing from one type to another.
Should I create a separate Jira for glob handling?
> Incorrect exit codes for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-6702
> URL: https://issues.apache.org/jira/browse/HADOOP-6702
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
> Reporter: Ravi Phulari
> Assignee: Ravi Phulari
> Priority: Minor
> Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> Currently incorrect exit codes are given for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
> This bug is due to missing update of errors count in {{FsShell.java}}.
> {code:title=FsShell.java|borderStyle=solid}
> int runCmdHandler(CmdHandler handler, String[] args,
> int startIndex, boolean recursive)
> throws IOException {
> int errors = 0;
>
> for (int i=startIndex; i<args.length; i++) {
> Path srcPath = new Path(args[i]);
> FileSystem srcFs = srcPath.getFileSystem(getConf());
> Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);
> for(Path path : paths) {
> try {
> FileStatus file = srcFs.getFileStatus(path);
> if (file == null) {
> System.err.println(handler.getName() +
> ": could not get status for '" + path + "'");
> errors++;
> } else {
> errors += runCmdHandler(handler, file, srcFs, recursive);
> }
> } catch (IOException e) {
> String msg = (e.getMessage() != null ? e.getLocalizedMessage() :
> (e.getCause().getMessage() != null ?
> e.getCause().getLocalizedMessage() : "null"));
> System.err.println(handler.getName() + ": could not get status for '"
> + path + "': " + msg.split("\n")[0]);
> errors++;
> }
> }
> }
> {code}
> If there are no files on HDFS matching to wildcard input then {{srcFs.globStatus(srcpath)}} returns 0.
> {{ Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);}}
> Resulting no increment in {{errors}} and command exits with 0 even though file/directory does not exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6702) Incorrect exit codes for "dfs
-chown", "dfs -chgrp" when input is given in wildcard format.
Posted by "Ravi Phulari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866248#action_12866248 ]
Ravi Phulari commented on HADOOP-6702:
--------------------------------------
bq.Should I create a separate Jira for glob handling?
Koji, could you please create Jira and explain desired fix for this error.
> Incorrect exit codes for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-6702
> URL: https://issues.apache.org/jira/browse/HADOOP-6702
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
> Reporter: Ravi Phulari
> Assignee: Ravi Phulari
> Priority: Minor
> Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> Currently incorrect exit codes are given for "dfs -chown", "dfs -chgrp" when input is given in wildcard format.
> This bug is due to missing update of errors count in {{FsShell.java}}.
> {code:title=FsShell.java|borderStyle=solid}
> int runCmdHandler(CmdHandler handler, String[] args,
> int startIndex, boolean recursive)
> throws IOException {
> int errors = 0;
>
> for (int i=startIndex; i<args.length; i++) {
> Path srcPath = new Path(args[i]);
> FileSystem srcFs = srcPath.getFileSystem(getConf());
> Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);
> for(Path path : paths) {
> try {
> FileStatus file = srcFs.getFileStatus(path);
> if (file == null) {
> System.err.println(handler.getName() +
> ": could not get status for '" + path + "'");
> errors++;
> } else {
> errors += runCmdHandler(handler, file, srcFs, recursive);
> }
> } catch (IOException e) {
> String msg = (e.getMessage() != null ? e.getLocalizedMessage() :
> (e.getCause().getMessage() != null ?
> e.getCause().getLocalizedMessage() : "null"));
> System.err.println(handler.getName() + ": could not get status for '"
> + path + "': " + msg.split("\n")[0]);
> errors++;
> }
> }
> }
> {code}
> If there are no files on HDFS matching to wildcard input then {{srcFs.globStatus(srcpath)}} returns 0.
> {{ Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);}}
> Resulting no increment in {{errors}} and command exits with 0 even though file/directory does not exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.