retain state between mappers

[ANKITBHATNAGAR <ab...@vantage.com>]

Hi All,

I am working on a task where I have to determine the count in the sequence
and increment by one.

My input to the job is multiple files
input/abc.csv
input/xyz.csv

So for example if my mapper is processing abc.csv I should be able to say my
current count is 0.

for file xyz.csv I should be able to say current count is 1.

is there a way I can retain the count between mappers and increment.?

Thanks
Ankit
-- 
View this message in context: http://old.nabble.com/retain-state-between-mappers-tp30851293p30851293.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: retain states between mappers

[Harsh J <qw...@gmail.com>]

If you know abc.csv is supposed to be "0", xyz supposed to be "1",
etc. then yes, there may be an easy way. For maintaining a persistent
count across jobs, one can even go that extra step to use another
service to take care of that.

If not, what would you do if multiple mappers get instantiated at the
same time for a lot of files? How would you determine the ordered
count for each if they all (or some, possibly) begin at the same time?

On Sat, Feb 5, 2011 at 5:33 PM, ANKITBHATNAGAR <ab...@vantage.com> wrote:
>
> Hi All,
>
> I am working on a task where I have to determine the count in the sequence
> and increment by one.
>
> My input to the job is multiple files
> input/abc.csv
> input/xyz.csv
>
> So for example if my mapper is processing abc.csv I should be able to say my
> current count is 0.
>
> for file xyz.csv I should be able to say current count is 1.
>
> is there a way I can retain the count between mappers and increment.?
>
> Thanks
> Ankit
> --
> View this message in context: http://old.nabble.com/retain-states-between-mappers-tp30851293p30851293.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>



-- 
Harsh J
www.harshj.com

Re: retain state between mappers

[Ted Dunning <td...@maprtech.com>]

Remember that mappers are not executed in a well defined order.

They can be executed in different order or even at the same time.  One
mapper can be run more than once.

There are two ways to get something like what you want, but the question you
asked is ill-posed.

First, you can adapt the input format so that it gives a different integer
to each split as a key.  This doesn't something like what you ask for since
each mapper will get a different integer and multiply executed mappers will
get the same key each time they are run.

Secondly, you could use central coordination server to act as a global
counter.  **THIS IS A REALLY BAD IDEA**  It is bad because it turns a
parallel computation into a partially sequential one and because it doesn't
account for the fact that mappers can be run multiple times.

On Sat, Feb 5, 2011 at 4:03 AM, ANKITBHATNAGAR <ab...@vantage.com>wrote:

> is there a way I can retain the count between mappers and increment.?
>