You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by neeral beladia <ne...@yahoo.com> on 2011/06/01 00:56:20 UTC
copyToLocal (from Amazon AWS)
Hi,
I am not sure if this question has been asked. Its more of a hadoop fs
question. I am trying to execute the following hadoop fs command :
hadoop fs -copyToLocal s3n://<Access Key>:<Secret Key>@<bucket
name>/file.txt /home/hadoop/workspace/file.txt
When I execute this command directly from the Terminal shell, it works
perfectly fine, however the above command from code doesn't execute. In fact,
it says :
Exception in thread "main" copyToLocal: null
Please note I am using Runtime.getRunTime().exec(cmdStr), where cmdStr is the
above hadoop command. Also, please note that hadoop fs -cp or hadoop fs -rmr
commands work fine with source and destination being both Amazon AWS locations.
In the above command (hadoop fs -copyToLocal) the destination is local location
to my machine(Ubuntu installed).
Your help would be greatly appreciated.
Thanks,
Neeral
Re: copyToLocal (from Amazon AWS)
Posted by Mapred Learn <ma...@gmail.com>.
Oops.. reading again, the command is working.
what is the exact string that you have in cmdStr ?
On Tue, May 31, 2011 at 4:51 PM, Mapred Learn <ma...@gmail.com>wrote:
> try using complete path for where you hadoop binary is present. For eg
> /usr/bin/hadoop instead of hadoop...
>
>
>
> On Tue, May 31, 2011 at 3:56 PM, neeral beladia <ne...@yahoo.com>wrote:
>
>> Hi,
>>
>> I am not sure if this question has been asked. Its more of a hadoop fs
>> question. I am trying to execute the following hadoop fs command :
>>
>> hadoop fs -copyToLocal s3n://<Access Key>:<Secret Key>@<bucket
>> name>/file.txt /home/hadoop/workspace/file.txt
>>
>> When I execute this command directly from the Terminal shell, it works
>> perfectly fine, however the above command from code doesn't execute. In
>> fact,
>> it says :
>>
>> Exception in thread "main" copyToLocal: null
>>
>> Please note I am using Runtime.getRunTime().exec(cmdStr), where cmdStr is
>> the
>> above hadoop command. Also, please note that hadoop fs -cp or hadoop fs
>> -rmr
>> commands work fine with source and destination being both Amazon AWS
>> locations.
>> In the above command (hadoop fs -copyToLocal) the destination is local
>> location
>> to my machine(Ubuntu installed).
>>
>> Your help would be greatly appreciated.
>>
>> Thanks,
>>
>> Neeral
>
>
>
Re: DistributedCache - getLocalCacheFiles method returns null
Posted by neeral beladia <ne...@yahoo.com>.
btw, just to let you know that I am running my job in a pseudo-distributed mode.
Thanks,
Neeral
________________________________
From: neeral beladia <ne...@yahoo.com>
To: common-user@hadoop.apache.org
Sent: Tue, May 31, 2011 10:00:00 PM
Subject: DistributedCache - getLocalCacheFiles method returns null
Hi,
I have a file on amazon aws under :
s3n://<Access Key>:<Secret Key>@<Bucket Name>/file.txt
I want this file to be accessible by the slave nodes via Distributed Cache.
I put the following after the job configuration statements in the Driver program
:
DistributedCache.addCacheFile(new Path("s3n://<Access Key>:<Secret Key>@<Bucket
Name>/file.txt").toUri(), job.getConfiguration());
Also in my setup method in the mapper class, I have the below statement :
Path[] cacheFiles =
DistributedCache.getLocalCacheFiles(context.getConfiguration());
cacheFiles is gettng assigned null.
Could you please let me know what I am doing wrong here ? The file does exist on
S3.
Thanks,
Neeral
DistributedCache - getLocalCacheFiles method returns null
Posted by neeral beladia <ne...@yahoo.com>.
Hi,
I have a file on amazon aws under :
s3n://<Access Key>:<Secret Key>@<Bucket Name>/file.txt
I want this file to be accessible by the slave nodes via Distributed Cache.
I put the following after the job configuration statements in the Driver program
:
DistributedCache.addCacheFile(new Path("s3n://<Access Key>:<Secret Key>@<Bucket
Name>/file.txt").toUri(), job.getConfiguration());
Also in my setup method in the mapper class, I have the below statement :
Path[] cacheFiles =
DistributedCache.getLocalCacheFiles(context.getConfiguration());
cacheFiles is gettng assigned null.
Could you please let me know what I am doing wrong here ? The file does exist on
S3.
Thanks,
Neeral
Re: copyToLocal (from Amazon AWS)
Posted by Mapred Learn <ma...@gmail.com>.
try using complete path for where you hadoop binary is present. For eg
/usr/bin/hadoop instead of hadoop...
On Tue, May 31, 2011 at 3:56 PM, neeral beladia <ne...@yahoo.com>wrote:
> Hi,
>
> I am not sure if this question has been asked. Its more of a hadoop fs
> question. I am trying to execute the following hadoop fs command :
>
> hadoop fs -copyToLocal s3n://<Access Key>:<Secret Key>@<bucket
> name>/file.txt /home/hadoop/workspace/file.txt
>
> When I execute this command directly from the Terminal shell, it works
> perfectly fine, however the above command from code doesn't execute. In
> fact,
> it says :
>
> Exception in thread "main" copyToLocal: null
>
> Please note I am using Runtime.getRunTime().exec(cmdStr), where cmdStr is
> the
> above hadoop command. Also, please note that hadoop fs -cp or hadoop fs
> -rmr
> commands work fine with source and destination being both Amazon AWS
> locations.
> In the above command (hadoop fs -copyToLocal) the destination is local
> location
> to my machine(Ubuntu installed).
>
> Your help would be greatly appreciated.
>
> Thanks,
>
> Neeral