You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by samir das mohapatra <sa...@gmail.com> on 2013/06/03 23:11:13 UTC
copyToLocal Failed inside the cleanup(.........) of Map task
Dear All,
Is there any way to copy the intermediate output file of the mapper
into local folder after each map task complete.
Right now I am using
FileSystem.copyToLocalFile(hdfsLocation,localLocation);
indiste the cleanup of mapper task , but it is failing .
Exception file not found.
But if I am giving same statement after the job complete in driver class
,it is working fine. that i dont want.
protected void cleanup(Context context){
FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
}
Note: I need to copy the inter mediate output of the mapper to local file
system just after each map task complete. I dont want any reducer.
If this is not the right solution then how to solve this type of
scenario.
Any help.............
regards,
samir.
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Mischa Tuffield <mi...@mmt.me.uk>.
When you are configuring your job in what most people refer to as their "Driver" class you can simply not set a Reducer and only set a Mapper.
// makes the job a map only one
job.setNumReduceTasks(0);
job.setMapperClass(MyFooMapper.class);
Mischa
On 3 Jun 2013, at 22:11, samir das mohapatra <sa...@gmail.com> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of scenario.
>
> Any help.............
>
> regards,
> samir.
_______________________________
Mischa Tuffield PhD
http://mmt.me.uk/
@mischat
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Shahab Yunus <sh...@gmail.com>.
Check out Pages 217-220 of the Hadoop: The Definitive Guide book. It has
some nice explanation. Also,
http://whiteycode.blogspot.com/2012/06/hadoop-removing-empty-output-files.html
Plus as Jay said, explanation of your use-case might also be helpful.
Regards,
Shahab
On Mon, Jun 3, 2013 at 5:44 PM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Do you have any link or example ? could you please send me ?
>
>
> On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <sh...@gmail.com>wrote:
>
>> Have you taken a look into extending the FileOutputFormat class and
>> overriding the OutputCommitter API functionality?
>>
>> Regards,
>> Shahab
>>
>>
>> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
>> samir.helpdoc@gmail.com> wrote:
>>
>>> Dear All,
>>>
>>> Is there any way to copy the intermediate output file of the mapper
>>> into local folder after each map task complete.
>>>
>>> Right now I am using
>>>
>>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>>> indiste the cleanup of mapper task , but it is failing .
>>>
>>> Exception file not found.
>>>
>>> But if I am giving same statement after the job complete in driver class
>>> ,it is working fine. that i dont want.
>>>
>>> protected void cleanup(Context context){
>>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>>> }
>>>
>>> Note: I need to copy the inter mediate output of the mapper to local
>>> file system just after each map task complete. I dont want any reducer.
>>>
>>> If this is not the right solution then how to solve this type of
>>> scenario.
>>>
>>> Any help.............
>>>
>>> regards,
>>> samir.
>>>
>>
>>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Shahab Yunus <sh...@gmail.com>.
Check out Pages 217-220 of the Hadoop: The Definitive Guide book. It has
some nice explanation. Also,
http://whiteycode.blogspot.com/2012/06/hadoop-removing-empty-output-files.html
Plus as Jay said, explanation of your use-case might also be helpful.
Regards,
Shahab
On Mon, Jun 3, 2013 at 5:44 PM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Do you have any link or example ? could you please send me ?
>
>
> On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <sh...@gmail.com>wrote:
>
>> Have you taken a look into extending the FileOutputFormat class and
>> overriding the OutputCommitter API functionality?
>>
>> Regards,
>> Shahab
>>
>>
>> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
>> samir.helpdoc@gmail.com> wrote:
>>
>>> Dear All,
>>>
>>> Is there any way to copy the intermediate output file of the mapper
>>> into local folder after each map task complete.
>>>
>>> Right now I am using
>>>
>>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>>> indiste the cleanup of mapper task , but it is failing .
>>>
>>> Exception file not found.
>>>
>>> But if I am giving same statement after the job complete in driver class
>>> ,it is working fine. that i dont want.
>>>
>>> protected void cleanup(Context context){
>>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>>> }
>>>
>>> Note: I need to copy the inter mediate output of the mapper to local
>>> file system just after each map task complete. I dont want any reducer.
>>>
>>> If this is not the right solution then how to solve this type of
>>> scenario.
>>>
>>> Any help.............
>>>
>>> regards,
>>> samir.
>>>
>>
>>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Shahab Yunus <sh...@gmail.com>.
Check out Pages 217-220 of the Hadoop: The Definitive Guide book. It has
some nice explanation. Also,
http://whiteycode.blogspot.com/2012/06/hadoop-removing-empty-output-files.html
Plus as Jay said, explanation of your use-case might also be helpful.
Regards,
Shahab
On Mon, Jun 3, 2013 at 5:44 PM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Do you have any link or example ? could you please send me ?
>
>
> On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <sh...@gmail.com>wrote:
>
>> Have you taken a look into extending the FileOutputFormat class and
>> overriding the OutputCommitter API functionality?
>>
>> Regards,
>> Shahab
>>
>>
>> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
>> samir.helpdoc@gmail.com> wrote:
>>
>>> Dear All,
>>>
>>> Is there any way to copy the intermediate output file of the mapper
>>> into local folder after each map task complete.
>>>
>>> Right now I am using
>>>
>>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>>> indiste the cleanup of mapper task , but it is failing .
>>>
>>> Exception file not found.
>>>
>>> But if I am giving same statement after the job complete in driver class
>>> ,it is working fine. that i dont want.
>>>
>>> protected void cleanup(Context context){
>>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>>> }
>>>
>>> Note: I need to copy the inter mediate output of the mapper to local
>>> file system just after each map task complete. I dont want any reducer.
>>>
>>> If this is not the right solution then how to solve this type of
>>> scenario.
>>>
>>> Any help.............
>>>
>>> regards,
>>> samir.
>>>
>>
>>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Shahab Yunus <sh...@gmail.com>.
Check out Pages 217-220 of the Hadoop: The Definitive Guide book. It has
some nice explanation. Also,
http://whiteycode.blogspot.com/2012/06/hadoop-removing-empty-output-files.html
Plus as Jay said, explanation of your use-case might also be helpful.
Regards,
Shahab
On Mon, Jun 3, 2013 at 5:44 PM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Do you have any link or example ? could you please send me ?
>
>
> On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <sh...@gmail.com>wrote:
>
>> Have you taken a look into extending the FileOutputFormat class and
>> overriding the OutputCommitter API functionality?
>>
>> Regards,
>> Shahab
>>
>>
>> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
>> samir.helpdoc@gmail.com> wrote:
>>
>>> Dear All,
>>>
>>> Is there any way to copy the intermediate output file of the mapper
>>> into local folder after each map task complete.
>>>
>>> Right now I am using
>>>
>>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>>> indiste the cleanup of mapper task , but it is failing .
>>>
>>> Exception file not found.
>>>
>>> But if I am giving same statement after the job complete in driver class
>>> ,it is working fine. that i dont want.
>>>
>>> protected void cleanup(Context context){
>>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>>> }
>>>
>>> Note: I need to copy the inter mediate output of the mapper to local
>>> file system just after each map task complete. I dont want any reducer.
>>>
>>> If this is not the right solution then how to solve this type of
>>> scenario.
>>>
>>> Any help.............
>>>
>>> regards,
>>> samir.
>>>
>>
>>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by samir das mohapatra <sa...@gmail.com>.
Do you have any link or example ? could you please send me ?
On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <sh...@gmail.com> wrote:
> Have you taken a look into extending the FileOutputFormat class and
> overriding the OutputCommitter API functionality?
>
> Regards,
> Shahab
>
>
> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
> samir.helpdoc@gmail.com> wrote:
>
>> Dear All,
>>
>> Is there any way to copy the intermediate output file of the mapper
>> into local folder after each map task complete.
>>
>> Right now I am using
>>
>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>> indiste the cleanup of mapper task , but it is failing .
>>
>> Exception file not found.
>>
>> But if I am giving same statement after the job complete in driver class
>> ,it is working fine. that i dont want.
>>
>> protected void cleanup(Context context){
>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>> }
>>
>> Note: I need to copy the inter mediate output of the mapper to local file
>> system just after each map task complete. I dont want any reducer.
>>
>> If this is not the right solution then how to solve this type of
>> scenario.
>>
>> Any help.............
>>
>> regards,
>> samir.
>>
>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by samir das mohapatra <sa...@gmail.com>.
Do you have any link or example ? could you please send me ?
On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <sh...@gmail.com> wrote:
> Have you taken a look into extending the FileOutputFormat class and
> overriding the OutputCommitter API functionality?
>
> Regards,
> Shahab
>
>
> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
> samir.helpdoc@gmail.com> wrote:
>
>> Dear All,
>>
>> Is there any way to copy the intermediate output file of the mapper
>> into local folder after each map task complete.
>>
>> Right now I am using
>>
>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>> indiste the cleanup of mapper task , but it is failing .
>>
>> Exception file not found.
>>
>> But if I am giving same statement after the job complete in driver class
>> ,it is working fine. that i dont want.
>>
>> protected void cleanup(Context context){
>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>> }
>>
>> Note: I need to copy the inter mediate output of the mapper to local file
>> system just after each map task complete. I dont want any reducer.
>>
>> If this is not the right solution then how to solve this type of
>> scenario.
>>
>> Any help.............
>>
>> regards,
>> samir.
>>
>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by samir das mohapatra <sa...@gmail.com>.
Do you have any link or example ? could you please send me ?
On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <sh...@gmail.com> wrote:
> Have you taken a look into extending the FileOutputFormat class and
> overriding the OutputCommitter API functionality?
>
> Regards,
> Shahab
>
>
> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
> samir.helpdoc@gmail.com> wrote:
>
>> Dear All,
>>
>> Is there any way to copy the intermediate output file of the mapper
>> into local folder after each map task complete.
>>
>> Right now I am using
>>
>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>> indiste the cleanup of mapper task , but it is failing .
>>
>> Exception file not found.
>>
>> But if I am giving same statement after the job complete in driver class
>> ,it is working fine. that i dont want.
>>
>> protected void cleanup(Context context){
>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>> }
>>
>> Note: I need to copy the inter mediate output of the mapper to local file
>> system just after each map task complete. I dont want any reducer.
>>
>> If this is not the right solution then how to solve this type of
>> scenario.
>>
>> Any help.............
>>
>> regards,
>> samir.
>>
>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by samir das mohapatra <sa...@gmail.com>.
Do you have any link or example ? could you please send me ?
On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <sh...@gmail.com> wrote:
> Have you taken a look into extending the FileOutputFormat class and
> overriding the OutputCommitter API functionality?
>
> Regards,
> Shahab
>
>
> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
> samir.helpdoc@gmail.com> wrote:
>
>> Dear All,
>>
>> Is there any way to copy the intermediate output file of the mapper
>> into local folder after each map task complete.
>>
>> Right now I am using
>>
>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>> indiste the cleanup of mapper task , but it is failing .
>>
>> Exception file not found.
>>
>> But if I am giving same statement after the job complete in driver class
>> ,it is working fine. that i dont want.
>>
>> protected void cleanup(Context context){
>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>> }
>>
>> Note: I need to copy the inter mediate output of the mapper to local file
>> system just after each map task complete. I dont want any reducer.
>>
>> If this is not the right solution then how to solve this type of
>> scenario.
>>
>> Any help.............
>>
>> regards,
>> samir.
>>
>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Shahab Yunus <sh...@gmail.com>.
Have you taken a look into extending the FileOutputFormat class and
overriding the OutputCommitter API functionality?
Regards,
Shahab
On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper
> into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class
> ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file
> system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of
> scenario.
>
> Any help.............
>
> regards,
> samir.
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Shahab Yunus <sh...@gmail.com>.
Have you taken a look into extending the FileOutputFormat class and
overriding the OutputCommitter API functionality?
Regards,
Shahab
On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper
> into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class
> ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file
> system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of
> scenario.
>
> Any help.............
>
> regards,
> samir.
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Deepak Vohra <dv...@yahoo.com>.
Samir,
The intermediate output of the mapper is already output to the local filesystem, not HDFS.
The temporary intermediate file path is
FileOutputFormat.getWorkOutputPath(context)
________________________________
From: samir das mohapatra <sa...@gmail.com>
To: user@hadoop.apache.org; user-help@hadoop.apache.org; cdh-user@cloudera.org; cdh-user@cloudera.com
Sent: Monday, June 3, 2013 2:11 PM
Subject: copyToLocal Failed inside the cleanup(.........) of Map task
Dear All,
Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
Right now I am using
FileSystem.copyToLocalFile(hdfsLocation,localLocation);
indiste the cleanup of mapper task , but it is failing .
Exception file not found.
But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
protected void cleanup(Context context){
FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
}
Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
If this is not the right solution then how to solve this type of scenario.
Any help.............
regards,
samir.
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Mischa Tuffield <mi...@mmt.me.uk>.
When you are configuring your job in what most people refer to as their "Driver" class you can simply not set a Reducer and only set a Mapper.
// makes the job a map only one
job.setNumReduceTasks(0);
job.setMapperClass(MyFooMapper.class);
Mischa
On 3 Jun 2013, at 22:11, samir das mohapatra <sa...@gmail.com> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of scenario.
>
> Any help.............
>
> regards,
> samir.
_______________________________
Mischa Tuffield PhD
http://mmt.me.uk/
@mischat
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Jay Vyas <ja...@gmail.com>.
Just for context and so that we can understand the question --- why are you needing to copy intermediate mapper output?
On Jun 3, 2013, at 4:11 PM, samir das mohapatra <sa...@gmail.com> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of scenario.
>
> Any help.............
>
> regards,
> samir.
> --
>
>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Deepak Vohra <dv...@yahoo.com>.
Samir,
The intermediate output of the mapper is already output to the local filesystem, not HDFS.
The temporary intermediate file path is
FileOutputFormat.getWorkOutputPath(context)
________________________________
From: samir das mohapatra <sa...@gmail.com>
To: user@hadoop.apache.org; user-help@hadoop.apache.org; cdh-user@cloudera.org; cdh-user@cloudera.com
Sent: Monday, June 3, 2013 2:11 PM
Subject: copyToLocal Failed inside the cleanup(.........) of Map task
Dear All,
Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
Right now I am using
FileSystem.copyToLocalFile(hdfsLocation,localLocation);
indiste the cleanup of mapper task , but it is failing .
Exception file not found.
But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
protected void cleanup(Context context){
FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
}
Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
If this is not the right solution then how to solve this type of scenario.
Any help.............
regards,
samir.
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Raj K Singh <ra...@gmail.com>.
by default hadoop keep intermediate values produced by mapper in the local
file system,you can get the the handle on it using
FileOutputFormat.getWorkOutputPath(context)
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Tue, Jun 4, 2013 at 2:41 AM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper
> into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class
> ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file
> system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of
> scenario.
>
> Any help.............
>
> regards,
> samir.
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Jay Vyas <ja...@gmail.com>.
Just for context and so that we can understand the question --- why are you needing to copy intermediate mapper output?
On Jun 3, 2013, at 4:11 PM, samir das mohapatra <sa...@gmail.com> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of scenario.
>
> Any help.............
>
> regards,
> samir.
> --
>
>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Deepak Vohra <dv...@yahoo.com>.
Samir,
The intermediate output of the mapper is already output to the local filesystem, not HDFS.
The temporary intermediate file path is
FileOutputFormat.getWorkOutputPath(context)
________________________________
From: samir das mohapatra <sa...@gmail.com>
To: user@hadoop.apache.org; user-help@hadoop.apache.org; cdh-user@cloudera.org; cdh-user@cloudera.com
Sent: Monday, June 3, 2013 2:11 PM
Subject: copyToLocal Failed inside the cleanup(.........) of Map task
Dear All,
Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
Right now I am using
FileSystem.copyToLocalFile(hdfsLocation,localLocation);
indiste the cleanup of mapper task , but it is failing .
Exception file not found.
But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
protected void cleanup(Context context){
FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
}
Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
If this is not the right solution then how to solve this type of scenario.
Any help.............
regards,
samir.
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Mischa Tuffield <mi...@mmt.me.uk>.
When you are configuring your job in what most people refer to as their "Driver" class you can simply not set a Reducer and only set a Mapper.
// makes the job a map only one
job.setNumReduceTasks(0);
job.setMapperClass(MyFooMapper.class);
Mischa
On 3 Jun 2013, at 22:11, samir das mohapatra <sa...@gmail.com> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of scenario.
>
> Any help.............
>
> regards,
> samir.
_______________________________
Mischa Tuffield PhD
http://mmt.me.uk/
@mischat
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Deepak Vohra <dv...@yahoo.com>.
Samir,
The intermediate output of the mapper is already output to the local filesystem, not HDFS.
The temporary intermediate file path is
FileOutputFormat.getWorkOutputPath(context)
________________________________
From: samir das mohapatra <sa...@gmail.com>
To: user@hadoop.apache.org; user-help@hadoop.apache.org; cdh-user@cloudera.org; cdh-user@cloudera.com
Sent: Monday, June 3, 2013 2:11 PM
Subject: copyToLocal Failed inside the cleanup(.........) of Map task
Dear All,
Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
Right now I am using
FileSystem.copyToLocalFile(hdfsLocation,localLocation);
indiste the cleanup of mapper task , but it is failing .
Exception file not found.
But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
protected void cleanup(Context context){
FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
}
Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
If this is not the right solution then how to solve this type of scenario.
Any help.............
regards,
samir.
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Mischa Tuffield <mi...@mmt.me.uk>.
When you are configuring your job in what most people refer to as their "Driver" class you can simply not set a Reducer and only set a Mapper.
// makes the job a map only one
job.setNumReduceTasks(0);
job.setMapperClass(MyFooMapper.class);
Mischa
On 3 Jun 2013, at 22:11, samir das mohapatra <sa...@gmail.com> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of scenario.
>
> Any help.............
>
> regards,
> samir.
_______________________________
Mischa Tuffield PhD
http://mmt.me.uk/
@mischat
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Shahab Yunus <sh...@gmail.com>.
Have you taken a look into extending the FileOutputFormat class and
overriding the OutputCommitter API functionality?
Regards,
Shahab
On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper
> into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class
> ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file
> system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of
> scenario.
>
> Any help.............
>
> regards,
> samir.
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Raj K Singh <ra...@gmail.com>.
by default hadoop keep intermediate values produced by mapper in the local
file system,you can get the the handle on it using
FileOutputFormat.getWorkOutputPath(context)
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Tue, Jun 4, 2013 at 2:41 AM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper
> into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class
> ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file
> system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of
> scenario.
>
> Any help.............
>
> regards,
> samir.
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Raj K Singh <ra...@gmail.com>.
by default hadoop keep intermediate values produced by mapper in the local
file system,you can get the the handle on it using
FileOutputFormat.getWorkOutputPath(context)
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Tue, Jun 4, 2013 at 2:41 AM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper
> into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class
> ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file
> system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of
> scenario.
>
> Any help.............
>
> regards,
> samir.
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Raj K Singh <ra...@gmail.com>.
by default hadoop keep intermediate values produced by mapper in the local
file system,you can get the the handle on it using
FileOutputFormat.getWorkOutputPath(context)
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Tue, Jun 4, 2013 at 2:41 AM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper
> into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class
> ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file
> system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of
> scenario.
>
> Any help.............
>
> regards,
> samir.
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Shahab Yunus <sh...@gmail.com>.
Have you taken a look into extending the FileOutputFormat class and
overriding the OutputCommitter API functionality?
Regards,
Shahab
On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper
> into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class
> ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file
> system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of
> scenario.
>
> Any help.............
>
> regards,
> samir.
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Jay Vyas <ja...@gmail.com>.
Just for context and so that we can understand the question --- why are you needing to copy intermediate mapper output?
On Jun 3, 2013, at 4:11 PM, samir das mohapatra <sa...@gmail.com> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of scenario.
>
> Any help.............
>
> regards,
> samir.
> --
>
>
>
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Posted by Jay Vyas <ja...@gmail.com>.
Just for context and so that we can understand the question --- why are you needing to copy intermediate mapper output?
On Jun 3, 2013, at 4:11 PM, samir das mohapatra <sa...@gmail.com> wrote:
> Dear All,
>
> Is there any way to copy the intermediate output file of the mapper into local folder after each map task complete.
>
> Right now I am using
>
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);
> indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file system just after each map task complete. I dont want any reducer.
>
> If this is not the right solution then how to solve this type of scenario.
>
> Any help.............
>
> regards,
> samir.
> --
>
>
>