You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Anfernee Xu <an...@gmail.com> on 2010/08/25 15:07:29 UTC
how to implement post-mapper processing
I'm new to Hadoop and I want to use it for my data processing. My
understanding is that each Split will be processed by a mapper task, so for
my application I have mapper in which I populate backend data store with
data from splits, after all splits are consumed, I want to run a piece of
code to post-processing the data stored in backend data store, is there any
clean way to do this?
Can I have the post-processing running only at the node which involed in
mapper phase? Since the number of splits may be less than number of nodes in
the cluster, so some nodes may not involve in the job, I do not want them
involved in this post-processing either.
Thanks for your help.
--
--Anfernee
Re: how to implement post-mapper processing
Posted by Anfernee Xu <an...@gmail.com>.
Yes, it works if the node only has a single split, if it has multiple,
that's still a problem since not all data has been processed.
On Wed, Aug 25, 2010 at 11:08 PM, David Rosenstrauch <da...@darose.net>wrote:
> On 08/25/2010 10:36 AM, Anfernee Xu wrote:
>
>> Thanks all for your help.
>>
>> The challenge is that suppose I have 4 datanodes in cluster, but for a
>> given
>> input, I have 2 splits, therefore only 2 nodes out of 4 will run M/R job,
>> say nodeA and nodeB, after the job completes, the data from input has been
>> stored in datastore on nodeA and nodeB, nodeC and nodeD are intact at this
>> moment, for now I need to run a post-processing on nodeA and nodeB to get
>> my
>> data ready, originally I think I can have another M/R job also with 2
>> splits, but I cannot tell which node will be selected to run these splits,
>> I
>> expected the same nodes will be selected.
>>
>> Anfernee
>>
>
> Well then you could put your post-processing in Mapper.cleanup.
>
>
> http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapreduce/Mapper.html#cleanup%28org.apache.hadoop.mapreduce.Mapper.Context%29
>
> DR
>
--
--Anfernee
Re: how to implement post-mapper processing
Posted by David Rosenstrauch <da...@darose.net>.
On 08/25/2010 10:36 AM, Anfernee Xu wrote:
> Thanks all for your help.
>
> The challenge is that suppose I have 4 datanodes in cluster, but for a given
> input, I have 2 splits, therefore only 2 nodes out of 4 will run M/R job,
> say nodeA and nodeB, after the job completes, the data from input has been
> stored in datastore on nodeA and nodeB, nodeC and nodeD are intact at this
> moment, for now I need to run a post-processing on nodeA and nodeB to get my
> data ready, originally I think I can have another M/R job also with 2
> splits, but I cannot tell which node will be selected to run these splits, I
> expected the same nodes will be selected.
>
> Anfernee
Well then you could put your post-processing in Mapper.cleanup.
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapreduce/Mapper.html#cleanup%28org.apache.hadoop.mapreduce.Mapper.Context%29
DR
Re: how to implement post-mapper processing
Posted by Anfernee Xu <an...@gmail.com>.
Thanks all for your help.
The challenge is that suppose I have 4 datanodes in cluster, but for a given
input, I have 2 splits, therefore only 2 nodes out of 4 will run M/R job,
say nodeA and nodeB, after the job completes, the data from input has been
stored in datastore on nodeA and nodeB, nodeC and nodeD are intact at this
moment, for now I need to run a post-processing on nodeA and nodeB to get my
data ready, originally I think I can have another M/R job also with 2
splits, but I cannot tell which node will be selected to run these splits, I
expected the same nodes will be selected.
Anfernee
On Wed, Aug 25, 2010 at 10:18 PM, David Rosenstrauch <da...@darose.net>wrote:
> On 08/25/2010 09:07 AM, Anfernee Xu wrote:
>
>> I'm new to Hadoop and I want to use it for my data processing. My
>> understanding is that each Split will be processed by a mapper task, so
>> for
>> my application I have mapper in which I populate backend data store with
>> data from splits, after all splits are consumed, I want to run a piece of
>> code to post-processing the data stored in backend data store, is there
>> any
>> clean way to do this?
>>
>> Can I have the post-processing running only at the node which involed in
>> mapper phase? Since the number of splits may be less than number of nodes
>> in
>> the cluster, so some nodes may not involve in the job, I do not want them
>> involved in this post-processing either.
>>
>> Thanks for your help.
>>
>
> Couldn't you just do a "submit and wait" on your map reduce job, and then
> have the app that's doing the submitting do a cleanup process after the job
> completes which performs the post-processing?
>
> DR
>
--
--Anfernee
Re: how to implement post-mapper processing
Posted by David Rosenstrauch <da...@darose.net>.
On 08/25/2010 09:07 AM, Anfernee Xu wrote:
> I'm new to Hadoop and I want to use it for my data processing. My
> understanding is that each Split will be processed by a mapper task, so for
> my application I have mapper in which I populate backend data store with
> data from splits, after all splits are consumed, I want to run a piece of
> code to post-processing the data stored in backend data store, is there any
> clean way to do this?
>
> Can I have the post-processing running only at the node which involed in
> mapper phase? Since the number of splits may be less than number of nodes in
> the cluster, so some nodes may not involve in the job, I do not want them
> involved in this post-processing either.
>
> Thanks for your help.
Couldn't you just do a "submit and wait" on your map reduce job, and
then have the app that's doing the submitting do a cleanup process after
the job completes which performs the post-processing?
DR
Re: how to implement post-mapper processing
Posted by Jesper Utoft <je...@gmail.com>.
Hello.
I do not know if i get this right.
But you have some files you split. Then want to process those.
Could a job with multiple mappers and a single reducer do a thing like this?
Cheers
Jesper
2010/8/25 Anfernee Xu <an...@gmail.com>
> Hi Ken,
>
> Thanks for your reply, but I do not think chainmapper is helperful for me,
> since the next mapper will process the current record(key/value pair)
> produced by current mapper in the chain, but I want the next mapper shoud
> run only after all records and all splits are processed.
>
> Anfernee
>
> On Wed, Aug 25, 2010 at 9:25 PM, Ken Goodhope <ke...@gmail.com>wrote:
>
>> Look into chainmapper. Sounds like it might satisfy your needs.
>>
>
>
>
> --
> --Anfernee
>
Re: how to implement post-mapper processing
Posted by Anfernee Xu <an...@gmail.com>.
Hi Ken,
Thanks for your reply, but I do not think chainmapper is helperful for me,
since the next mapper will process the current record(key/value pair)
produced by current mapper in the chain, but I want the next mapper shoud
run only after all records and all splits are processed.
Anfernee
On Wed, Aug 25, 2010 at 9:25 PM, Ken Goodhope <ke...@gmail.com> wrote:
> Look into chainmapper. Sounds like it might satisfy your needs.
>
--
--Anfernee
Re: how to implement post-mapper processing
Posted by Ken Goodhope <ke...@gmail.com>.
Look into chainmapper. Sounds like it might satisfy your needs.