You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mark Desnoyer <md...@gmail.com> on 2009/06/18 16:12:02 UTC
Getting Task ID inside a Mapper
Hi,
I was wondering if it's possible to get a hold of the task id inside a
mapper? I cant' seem to find a way by trolling through the API reference.
I'm trying to implement a Map Reduce version of Latent Dirichlet Allocation
and I need to be able to initialize a random number generator in a task
specific way so that if the task fails and is rerun elsewhere, the results
are the same. Thanks in advance.
Cheers,
Mark Desnoyer
Re: Getting Task ID inside a Mapper
Posted by jason hadoop <ja...@gmail.com>.
The task id is readily available, if you override the configure method.
The MapReduceBase class in the Pro Hadoop Book examples does this and makes
the taskId available as a class field.
On Thu, Jun 18, 2009 at 7:33 AM, Mark Desnoyer <md...@gmail.com> wrote:
> Thanks! I'll try that.
>
> -Mark
>
> On Thu, Jun 18, 2009 at 10:27 AM, Jingkei Ly <Ji...@detica.com>
> wrote:
>
> > I think you can use job.getInt("mapred.task.partition",-1) to get the
> > mapper ID, which should be the same for the mapper across task reruns.
> >
> > -----Original Message-----
> > From: Piotr Praczyk [mailto:piotr.praczyk@gmail.com]
> > Sent: 18 June 2009 15:19
> > To: core-user@hadoop.apache.org
> > Subject: Re: Getting Task ID inside a Mapper
> >
> > Hi
> > Why don't you provide a seed of random generator generated outside the
> > task
> > ? Then when the task fails, you can provide the same value stored
> > somewhere
> > outside.
> > You could use the task configuration to do so.
> >
> > I don't know anything about obtaining the task ID from within.
> >
> >
> > regards
> > Piotr
> >
> > 2009/6/18 Mark Desnoyer <md...@gmail.com>
> >
> > > Hi,
> > >
> > > I was wondering if it's possible to get a hold of the task id inside a
> > > mapper? I cant' seem to find a way by trolling through the API
> > reference.
> > > I'm trying to implement a Map Reduce version of Latent Dirichlet
> > Allocation
> > > and I need to be able to initialize a random number generator in a
> > task
> > > specific way so that if the task fails and is rerun elsewhere, the
> > results
> > > are the same. Thanks in advance.
> > >
> > > Cheers,
> > > Mark Desnoyer
> > >
> >
> >
> >
> > This message should be regarded as confidential. If you have received
> this
> > email in error please notify the sender and destroy it immediately.
> > Statements of intent shall only become binding when confirmed in hard
> copy
> > by an authorised signatory. The contents of this email may relate to
> > dealings with other companies within the Detica Group plc group of
> > companies.
> >
> > Detica Limited is registered in England under No: 1337451.
> >
> > Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
> > England.
> >
> >
> >
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Re: Getting Task ID inside a Mapper
Posted by Mark Desnoyer <md...@gmail.com>.
Thanks! I'll try that.
-Mark
On Thu, Jun 18, 2009 at 10:27 AM, Jingkei Ly <Ji...@detica.com> wrote:
> I think you can use job.getInt("mapred.task.partition",-1) to get the
> mapper ID, which should be the same for the mapper across task reruns.
>
> -----Original Message-----
> From: Piotr Praczyk [mailto:piotr.praczyk@gmail.com]
> Sent: 18 June 2009 15:19
> To: core-user@hadoop.apache.org
> Subject: Re: Getting Task ID inside a Mapper
>
> Hi
> Why don't you provide a seed of random generator generated outside the
> task
> ? Then when the task fails, you can provide the same value stored
> somewhere
> outside.
> You could use the task configuration to do so.
>
> I don't know anything about obtaining the task ID from within.
>
>
> regards
> Piotr
>
> 2009/6/18 Mark Desnoyer <md...@gmail.com>
>
> > Hi,
> >
> > I was wondering if it's possible to get a hold of the task id inside a
> > mapper? I cant' seem to find a way by trolling through the API
> reference.
> > I'm trying to implement a Map Reduce version of Latent Dirichlet
> Allocation
> > and I need to be able to initialize a random number generator in a
> task
> > specific way so that if the task fails and is rerun elsewhere, the
> results
> > are the same. Thanks in advance.
> >
> > Cheers,
> > Mark Desnoyer
> >
>
>
>
> This message should be regarded as confidential. If you have received this
> email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy
> by an authorised signatory. The contents of this email may relate to
> dealings with other companies within the Detica Group plc group of
> companies.
>
> Detica Limited is registered in England under No: 1337451.
>
> Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
> England.
>
>
>
RE: Getting Task ID inside a Mapper
Posted by Jingkei Ly <Ji...@detica.com>.
I think you can use job.getInt("mapred.task.partition",-1) to get the
mapper ID, which should be the same for the mapper across task reruns.
-----Original Message-----
From: Piotr Praczyk [mailto:piotr.praczyk@gmail.com]
Sent: 18 June 2009 15:19
To: core-user@hadoop.apache.org
Subject: Re: Getting Task ID inside a Mapper
Hi
Why don't you provide a seed of random generator generated outside the
task
? Then when the task fails, you can provide the same value stored
somewhere
outside.
You could use the task configuration to do so.
I don't know anything about obtaining the task ID from within.
regards
Piotr
2009/6/18 Mark Desnoyer <md...@gmail.com>
> Hi,
>
> I was wondering if it's possible to get a hold of the task id inside a
> mapper? I cant' seem to find a way by trolling through the API
reference.
> I'm trying to implement a Map Reduce version of Latent Dirichlet
Allocation
> and I need to be able to initialize a random number generator in a
task
> specific way so that if the task fails and is rerun elsewhere, the
results
> are the same. Thanks in advance.
>
> Cheers,
> Mark Desnoyer
>
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Group plc group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.
Re: Getting Task ID inside a Mapper
Posted by Piotr Praczyk <pi...@gmail.com>.
Hi
Why don't you provide a seed of random generator generated outside the task
? Then when the task fails, you can provide the same value stored somewhere
outside.
You could use the task configuration to do so.
I don't know anything about obtaining the task ID from within.
regards
Piotr
2009/6/18 Mark Desnoyer <md...@gmail.com>
> Hi,
>
> I was wondering if it's possible to get a hold of the task id inside a
> mapper? I cant' seem to find a way by trolling through the API reference.
> I'm trying to implement a Map Reduce version of Latent Dirichlet Allocation
> and I need to be able to initialize a random number generator in a task
> specific way so that if the task fails and is rerun elsewhere, the results
> are the same. Thanks in advance.
>
> Cheers,
> Mark Desnoyer
>