You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Kohsuke Kawaguchi (JIRA)" <ji...@apache.org> on 2009/03/15 18:22:50 UTC
[jira] Created: (HADOOP-5496) "du" fails on Cygwin
"du" fails on Cygwin
--------------------
Key: HADOOP-5496
URL: https://issues.apache.org/jira/browse/HADOOP-5496
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.19.1
Environment: Windows + Cygwin
Reporter: Kohsuke Kawaguchi
When I try to run a datanode on Windows, I get the following exception:
{noformat}
java.io.IOException: Expecting a line not the end of stream
at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DU.<init>(DU.java:53)
at org.apache.hadoop.fs.DU.<init>(DU.java:63)
at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
{noformat}
This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.
{noformat}
C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
du -sk C:\tmp\hadoop-SYSTEM\dfs\data
du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
{noformat}
For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.
Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5496) "du" fails on Cygwin
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting reassigned HADOOP-5496:
------------------------------------
Assignee: Kohsuke Kawaguchi
> The patch for this can be found on http://github.com/kohsuke [ ...]
Can you please attach your patch directly to this issue in Jira, checking the license checkbox if you wish your work to be includable in an Apache release. Thanks!
> "du" fails on Cygwin
> --------------------
>
> Key: HADOOP-5496
> URL: https://issues.apache.org/jira/browse/HADOOP-5496
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.1
> Environment: Windows + Cygwin
> Reporter: Kohsuke Kawaguchi
> Assignee: Kohsuke Kawaguchi
>
> When I try to run a datanode on Windows, I get the following exception:
> {noformat}
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
> at org.apache.hadoop.util.Shell.run(Shell.java:134)
> at org.apache.hadoop.fs.DU.<init>(DU.java:53)
> at org.apache.hadoop.fs.DU.<init>(DU.java:63)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
> {noformat}
> This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.
> {noformat}
> C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
> du -sk C:\tmp\hadoop-SYSTEM\dfs\data
> du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
> {noformat}
> For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.
> Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5496) "du" fails on Cygwin
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682427#action_12682427 ]
Doug Cutting commented on HADOOP-5496:
--------------------------------------
Does the TestDU unit test fail on Windows without this? If not, can we easily modify it so that it does?
> "du" fails on Cygwin
> --------------------
>
> Key: HADOOP-5496
> URL: https://issues.apache.org/jira/browse/HADOOP-5496
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.1
> Environment: Windows + Cygwin
> Reporter: Kohsuke Kawaguchi
> Assignee: Kohsuke Kawaguchi
> Attachments: HADOOP-5486
>
>
> When I try to run a datanode on Windows, I get the following exception:
> {noformat}
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
> at org.apache.hadoop.util.Shell.run(Shell.java:134)
> at org.apache.hadoop.fs.DU.<init>(DU.java:53)
> at org.apache.hadoop.fs.DU.<init>(DU.java:63)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
> {noformat}
> This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.
> {noformat}
> C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
> du -sk C:\tmp\hadoop-SYSTEM\dfs\data
> du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
> {noformat}
> For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.
> Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5496) "du" fails on Cygwin
Posted by "Kohsuke Kawaguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kohsuke Kawaguchi updated HADOOP-5496:
--------------------------------------
Attachment: HADOOP-5486
Patch
> "du" fails on Cygwin
> --------------------
>
> Key: HADOOP-5496
> URL: https://issues.apache.org/jira/browse/HADOOP-5496
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.1
> Environment: Windows + Cygwin
> Reporter: Kohsuke Kawaguchi
> Assignee: Kohsuke Kawaguchi
> Attachments: HADOOP-5486
>
>
> When I try to run a datanode on Windows, I get the following exception:
> {noformat}
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
> at org.apache.hadoop.util.Shell.run(Shell.java:134)
> at org.apache.hadoop.fs.DU.<init>(DU.java:53)
> at org.apache.hadoop.fs.DU.<init>(DU.java:63)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
> {noformat}
> This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.
> {noformat}
> C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
> du -sk C:\tmp\hadoop-SYSTEM\dfs\data
> du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
> {noformat}
> For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.
> Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5496) "du" fails on Cygwin
Posted by "Kohsuke Kawaguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682375#action_12682375 ]
Kohsuke Kawaguchi commented on HADOOP-5496:
-------------------------------------------
The patch for this can be found on http://github.com/kohsuke/hadoop/commit/27273cd0515e598b51eed48a1d0c87fc16ca995f
> "du" fails on Cygwin
> --------------------
>
> Key: HADOOP-5496
> URL: https://issues.apache.org/jira/browse/HADOOP-5496
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.1
> Environment: Windows + Cygwin
> Reporter: Kohsuke Kawaguchi
>
> When I try to run a datanode on Windows, I get the following exception:
> {noformat}
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
> at org.apache.hadoop.util.Shell.run(Shell.java:134)
> at org.apache.hadoop.fs.DU.<init>(DU.java:53)
> at org.apache.hadoop.fs.DU.<init>(DU.java:63)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
> at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
> {noformat}
> This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.
> {noformat}
> C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
> du -sk C:\tmp\hadoop-SYSTEM\dfs\data
> du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
> {noformat}
> For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.
> Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.