You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Anit Alexander <an...@gmail.com> on 2012/09/10 13:31:41 UTC
how to skip a mapper
Hello list,
Is it possible to start the mapper from a particular byte
location in a file which is in hdfs?
Regards,
Anit
Re: how to skip a mapper
Posted by Abhishek Shivkumar <ab...@gmail.com>.
When the input to the mapper is a key,value pair, the key is the byte
offset of the file contents.
So, may be we can check if the file byte offset meets your criterion to do
the mapper task or not.
Thank you!
With Regards,
Abhishek S
On Mon, Sep 10, 2012 at 5:04 PM, Michael Segel <mi...@hotmail.com>wrote:
> Maybe.
> It depends on what you're trying to do.
>
> On Sep 10, 2012, at 6:31 AM, Anit Alexander <an...@gmail.com> wrote:
>
> > Hello list,
> >
> > Is it possible to start the mapper from a particular byte
> > location in a file which is in hdfs?
> >
> > Regards,
> > Anit
> >
>
>
Re: how to skip a mapper
Posted by Abhishek Shivkumar <ab...@gmail.com>.
When the input to the mapper is a key,value pair, the key is the byte
offset of the file contents.
So, may be we can check if the file byte offset meets your criterion to do
the mapper task or not.
Thank you!
With Regards,
Abhishek S
On Mon, Sep 10, 2012 at 5:04 PM, Michael Segel <mi...@hotmail.com>wrote:
> Maybe.
> It depends on what you're trying to do.
>
> On Sep 10, 2012, at 6:31 AM, Anit Alexander <an...@gmail.com> wrote:
>
> > Hello list,
> >
> > Is it possible to start the mapper from a particular byte
> > location in a file which is in hdfs?
> >
> > Regards,
> > Anit
> >
>
>
Re: how to skip a mapper
Posted by Abhishek Shivkumar <ab...@gmail.com>.
When the input to the mapper is a key,value pair, the key is the byte
offset of the file contents.
So, may be we can check if the file byte offset meets your criterion to do
the mapper task or not.
Thank you!
With Regards,
Abhishek S
On Mon, Sep 10, 2012 at 5:04 PM, Michael Segel <mi...@hotmail.com>wrote:
> Maybe.
> It depends on what you're trying to do.
>
> On Sep 10, 2012, at 6:31 AM, Anit Alexander <an...@gmail.com> wrote:
>
> > Hello list,
> >
> > Is it possible to start the mapper from a particular byte
> > location in a file which is in hdfs?
> >
> > Regards,
> > Anit
> >
>
>
Re: how to skip a mapper
Posted by Abhishek Shivkumar <ab...@gmail.com>.
When the input to the mapper is a key,value pair, the key is the byte
offset of the file contents.
So, may be we can check if the file byte offset meets your criterion to do
the mapper task or not.
Thank you!
With Regards,
Abhishek S
On Mon, Sep 10, 2012 at 5:04 PM, Michael Segel <mi...@hotmail.com>wrote:
> Maybe.
> It depends on what you're trying to do.
>
> On Sep 10, 2012, at 6:31 AM, Anit Alexander <an...@gmail.com> wrote:
>
> > Hello list,
> >
> > Is it possible to start the mapper from a particular byte
> > location in a file which is in hdfs?
> >
> > Regards,
> > Anit
> >
>
>
Re: how to skip a mapper
Posted by Michael Segel <mi...@hotmail.com>.
Maybe.
It depends on what you're trying to do.
On Sep 10, 2012, at 6:31 AM, Anit Alexander <an...@gmail.com> wrote:
> Hello list,
>
> Is it possible to start the mapper from a particular byte
> location in a file which is in hdfs?
>
> Regards,
> Anit
>
Re: how to skip a mapper
Posted by Harsh J <ha...@cloudera.com>.
Anit,
Yes this is possible (and actually does happen in regular MR scenario
anyway - when the input is split across several locations). You'll
need a custom InputFormat#getSplits implementation to do this (create
input splits with the first offset itself set to the known offset
location, instead of 0).
On Mon, Sep 10, 2012 at 5:01 PM, Anit Alexander <an...@gmail.com> wrote:
> Hello list,
>
> Is it possible to start the mapper from a particular byte
> location in a file which is in hdfs?
>
> Regards,
> Anit
--
Harsh J
Re: how to skip a mapper
Posted by Michael Segel <mi...@hotmail.com>.
Maybe.
It depends on what you're trying to do.
On Sep 10, 2012, at 6:31 AM, Anit Alexander <an...@gmail.com> wrote:
> Hello list,
>
> Is it possible to start the mapper from a particular byte
> location in a file which is in hdfs?
>
> Regards,
> Anit
>
Re: how to skip a mapper
Posted by Harsh J <ha...@cloudera.com>.
Anit,
Yes this is possible (and actually does happen in regular MR scenario
anyway - when the input is split across several locations). You'll
need a custom InputFormat#getSplits implementation to do this (create
input splits with the first offset itself set to the known offset
location, instead of 0).
On Mon, Sep 10, 2012 at 5:01 PM, Anit Alexander <an...@gmail.com> wrote:
> Hello list,
>
> Is it possible to start the mapper from a particular byte
> location in a file which is in hdfs?
>
> Regards,
> Anit
--
Harsh J
Re: how to skip a mapper
Posted by Harsh J <ha...@cloudera.com>.
Anit,
Yes this is possible (and actually does happen in regular MR scenario
anyway - when the input is split across several locations). You'll
need a custom InputFormat#getSplits implementation to do this (create
input splits with the first offset itself set to the known offset
location, instead of 0).
On Mon, Sep 10, 2012 at 5:01 PM, Anit Alexander <an...@gmail.com> wrote:
> Hello list,
>
> Is it possible to start the mapper from a particular byte
> location in a file which is in hdfs?
>
> Regards,
> Anit
--
Harsh J
Re: how to skip a mapper
Posted by Harsh J <ha...@cloudera.com>.
Anit,
Yes this is possible (and actually does happen in regular MR scenario
anyway - when the input is split across several locations). You'll
need a custom InputFormat#getSplits implementation to do this (create
input splits with the first offset itself set to the known offset
location, instead of 0).
On Mon, Sep 10, 2012 at 5:01 PM, Anit Alexander <an...@gmail.com> wrote:
> Hello list,
>
> Is it possible to start the mapper from a particular byte
> location in a file which is in hdfs?
>
> Regards,
> Anit
--
Harsh J
Re: how to skip a mapper
Posted by Michael Segel <mi...@hotmail.com>.
Maybe.
It depends on what you're trying to do.
On Sep 10, 2012, at 6:31 AM, Anit Alexander <an...@gmail.com> wrote:
> Hello list,
>
> Is it possible to start the mapper from a particular byte
> location in a file which is in hdfs?
>
> Regards,
> Anit
>
Re: how to skip a mapper
Posted by Michael Segel <mi...@hotmail.com>.
Maybe.
It depends on what you're trying to do.
On Sep 10, 2012, at 6:31 AM, Anit Alexander <an...@gmail.com> wrote:
> Hello list,
>
> Is it possible to start the mapper from a particular byte
> location in a file which is in hdfs?
>
> Regards,
> Anit
>