You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by S D <sd...@gmail.com> on 2009/02/23 14:02:22 UTC

Can anyone verify Hadoop FS shell command return codes?

I'm attempting to use Hadoop FS shell (http://hadoop
.apache.org/core/docs/current/hdfs_shell.html) within a ruby script. My
challenge is that I'm unable to get the function return value of the
commands I'm invoking. As an example, I try to run get as follows

hadoop fs -get /user/hadoop/testFile.txt .

>From the command line this generally works but I need to be able to verify
that it is working during execution in my ruby script. The command should
return 0 on success and -1 on error. Based on

http://pasadenarb.com/2007/03/ruby-shell-commands.html

I am using backticks to make the hadoop call and get the return value. Here
is a dialogue within irb (Ruby's interactive shell) in which the command was
not successful:

irb(main):001:0> `hadoop dfs -get testFile.txt .`
get: null
=> ""

and a dialogue within irb in which the command was successful

irb(main):010:0> `hadoop dfs -get testFile.txt .`
=> ""

In both cases, neither a 0 nor a 1 appeared as a return value; indeed
nothing was returned. Can anyone who is using the FS command shell return
values within any scripting language (Ruby, PHP, Perl, ...) please confirm
that it is working as expected or send an example snippet?

Thanks,
John

Re: Can anyone verify Hadoop FS shell command return codes?

Posted by S D <sd...@gmail.com>.

Thanks for your response, Roldano. I decided to follow your suggestion with
Perl to remove the possibility of my misunderstanding Ruby syntax. I tried
the following Perl script

#!/usr/bin/perl
$var = system("hadoop dfs -test <param>");
print "\"$var\"";

Here are my results for the given value of <param>:

*Test           Value of <param>                                 Result*
1                -d <name of existing directory>                    0
2                -d <name of existing file>                           256
3                -d <name non-existing directory>               65280
4                -e <name of existing directory>                    0
5                -e <name of existing file>                             0
6                -e <name of non-existing file>                     256

In the above, "non-existing" means I used a file/directory name that was not
equivalent to any of the files/directories in the file system. The results
for tests 4 and 5 are what I would expect. The result for test 6 is a bit
unsatisfying but I guess it doesn't violate the contract. My concern is with
the result for tests 1-3. The result for test 1 should be "1" and the result
for tests 2 and 3 should both be "0". I'm basing this on

http://hadoop.apache.org/core/docs/current/hdfs_shell.html

Has anyone else used the Hadoop FS shell and got these results or are you
having better luck? Am I missing something here?

Note that I tried this with Hadoop 0.19.0 with both S3 Native and HDFS.

Thanks,
John



On Mon, Feb 23, 2009 at 11:43 AM, Roldano Cattoni <ca...@fbk.eu> wrote:

> You should distinguish between the output of a command and the return value
> of the command: usually they are captured in different ways by the
> interpreters (scripting languages or shells).
>
> For example:
>
> 1) in perl the return value is captured by using the system function:
>   $rv = system("cmd");
> so the $rv variable contains the value returned by cmd.
> Instead, with backticks you get the output of cmd:
>   $out = `cmd`;
>
> 2) in shells (sh/bash/tcsh) the return value is stored in the variable $?
> (dollar char followed by question-mark char).
> Instead, the output is again obtained with backticks.
>
> I don't know the way in which irb captures the return value: for analogy I
> would say that backticks are used for capturing the output even in irb.
>
> Best
>
>  Roldano
>
>
>
> On Mon, Feb 23, 2009 at 02:02:22PM +0100, S D wrote:
> > I'm attempting to use Hadoop FS shell (http://hadoop
> > .apache.org/core/docs/current/hdfs_shell.html) within a ruby script. My
> > challenge is that I'm unable to get the function return value of the
> > commands I'm invoking. As an example, I try to run get as follows
> >
> > hadoop fs -get /user/hadoop/testFile.txt .
> >
> > From the command line this generally works but I need to be able to
> verify
> > that it is working during execution in my ruby script. The command should
> > return 0 on success and -1 on error. Based on
> >
> > http://pasadenarb.com/2007/03/ruby-shell-commands.html
> >
> > I am using backticks to make the hadoop call and get the return value.
> Here
> > is a dialogue within irb (Ruby's interactive shell) in which the command
> was
> > not successful:
> >
> > irb(main):001:0> `hadoop dfs -get testFile.txt .`
> > get: null
> > => ""
> >
> > and a dialogue within irb in which the command was successful
> >
> > irb(main):010:0> `hadoop dfs -get testFile.txt .`
> > => ""
> >
> > In both cases, neither a 0 nor a 1 appeared as a return value; indeed
> > nothing was returned. Can anyone who is using the FS command shell return
> > values within any scripting language (Ruby, PHP, Perl, ...) please
> confirm
> > that it is working as expected or send an example snippet?
> >
> > Thanks,
> > John
>

Re: Can anyone verify Hadoop FS shell command return codes?

Posted by Roldano Cattoni <ca...@fbk.eu>.

You should distinguish between the output of a command and the return value
of the command: usually they are captured in different ways by the
interpreters (scripting languages or shells).

For example:

1) in perl the return value is captured by using the system function:
   $rv = system("cmd");
so the $rv variable contains the value returned by cmd. 
Instead, with backticks you get the output of cmd:
   $out = `cmd`;

2) in shells (sh/bash/tcsh) the return value is stored in the variable $?
(dollar char followed by question-mark char).
Instead, the output is again obtained with backticks.

I don't know the way in which irb captures the return value: for analogy I
would say that backticks are used for capturing the output even in irb.

Best

  Roldano

On Mon, Feb 23, 2009 at 02:02:22PM +0100, S D wrote:
> I'm attempting to use Hadoop FS shell (http://hadoop
> .apache.org/core/docs/current/hdfs_shell.html) within a ruby script. My
> challenge is that I'm unable to get the function return value of the
> commands I'm invoking. As an example, I try to run get as follows
> 
> hadoop fs -get /user/hadoop/testFile.txt .
> 
> From the command line this generally works but I need to be able to verify
> that it is working during execution in my ruby script. The command should
> return 0 on success and -1 on error. Based on
> 
> http://pasadenarb.com/2007/03/ruby-shell-commands.html
> 
> I am using backticks to make the hadoop call and get the return value. Here
> is a dialogue within irb (Ruby's interactive shell) in which the command was
> not successful:
> 
> irb(main):001:0> `hadoop dfs -get testFile.txt .`
> get: null
> => ""
> 
> and a dialogue within irb in which the command was successful
> 
> irb(main):010:0> `hadoop dfs -get testFile.txt .`
> => ""
> 
> In both cases, neither a 0 nor a 1 appeared as a return value; indeed
> nothing was returned. Can anyone who is using the FS command shell return
> values within any scripting language (Ruby, PHP, Perl, ...) please confirm
> that it is working as expected or send an example snippet?
> 
> Thanks,
> John

Re: Can anyone verify Hadoop FS shell command return codes?

Posted by Mikhail Yakshin <gr...@gmail.com>.

On Mon, Feb 23, 2009 at 4:02 PM, S D wrote:
> I'm attempting to use Hadoop FS shell (http://hadoop
> .apache.org/core/docs/current/hdfs_shell.html) within a ruby script. My
> challenge is that I'm unable to get the function return value of the
> commands I'm invoking. As an example, I try to run get as follows
>
> hadoop fs -get /user/hadoop/testFile.txt .
>
> From the command line this generally works but I need to be able to verify
> that it is working during execution in my ruby script. The command should
> return 0 on success and -1 on error. Based on
>
> http://pasadenarb.com/2007/03/ruby-shell-commands.html
>
> I am using backticks to make the hadoop call and get the return value. Here
> is a dialogue within irb (Ruby's interactive shell) in which the command was
> not successful:
>
> irb(main):001:0> `hadoop dfs -get testFile.txt .`
> get: null
> => ""
>
> and a dialogue within irb in which the command was successful
>
> irb(main):010:0> `hadoop dfs -get testFile.txt .`
> => ""
>
> In both cases, neither a 0 nor a 1 appeared as a return value; indeed
> nothing was returned. Can anyone who is using the FS command shell return
> values within any scripting language (Ruby, PHP, Perl, ...) please confirm
> that it is working as expected or send an example snippet?

You seem to confuse captured output of stdout and exit status. Try
analyzing $?.exitstatus in Ruby:

irb(main):001:0> `true`
=> ""
irb(main):002:0> $?.exitstatus
=> 0
irb(main):003:0> `false`
=> ""
irb(main):004:0> $?.exitstatus
=> 1

-- 
WBR, Mikhail Yakshin