You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Nick Cen <ce...@gmail.com> on 2009/02/22 09:55:04 UTC
How to use JobConf.setKeyFieldPartitionerOptions() method
Hi All,
Assume the output key from the mapper has the format "k1,k2", what i wanna
to do is to use the k1 instead the whole key to partition the output, what
parameter value shoud i provide to the setKeyFieldPartitionerOptions(). i
have try "-k 1", but it throuth an ArrayIndexOutOfBound Exception, Thanks in
advance.
--
http://daily.appspot.com/food/
Re: How to use JobConf.setKeyFieldPartitionerOptions() method
Posted by Nick Cen <ce...@gmail.com>.
Thanks Jason, It works.
2009/2/23 jason hadoop <ja...@gmail.com>
> For reasons that are not clear, in 19, the partitioner steps one character
> past the end of the field unless you are very explicit in your key
> specification.
> One would assume that -k2 would pick up the second token, even if it was
> the
> last field in the key, but -k2,2 is required
>
> As near as I can tell the -kX syntax means piece X including the separator
> character, which will of course not be present, if this is the last piece.
>
> In your case, try *setKeyFieldPartitionerOptions("-k 1,1");* I believe it
> will work.
>
>
> On Sun, Feb 22, 2009 at 12:55 AM, Nick Cen <ce...@gmail.com> wrote:
>
> > Hi All,
> >
> > Assume the output key from the mapper has the format "k1,k2", what i
> wanna
> > to do is to use the k1 instead the whole key to partition the output,
> what
> > parameter value shoud i provide to the setKeyFieldPartitionerOptions(). i
> > have try "-k 1", but it throuth an ArrayIndexOutOfBound Exception, Thanks
> > in
> > advance.
> >
> > --
> > http://daily.appspot.com/food/
> >
>
--
http://daily.appspot.com/food/
Re: How to use JobConf.setKeyFieldPartitionerOptions() method
Posted by jason hadoop <ja...@gmail.com>.
For reasons that are not clear, in 19, the partitioner steps one character
past the end of the field unless you are very explicit in your key
specification.
One would assume that -k2 would pick up the second token, even if it was the
last field in the key, but -k2,2 is required
As near as I can tell the -kX syntax means piece X including the separator
character, which will of course not be present, if this is the last piece.
In your case, try *setKeyFieldPartitionerOptions("-k 1,1");* I believe it
will work.
On Sun, Feb 22, 2009 at 12:55 AM, Nick Cen <ce...@gmail.com> wrote:
> Hi All,
>
> Assume the output key from the mapper has the format "k1,k2", what i wanna
> to do is to use the k1 instead the whole key to partition the output, what
> parameter value shoud i provide to the setKeyFieldPartitionerOptions(). i
> have try "-k 1", but it throuth an ArrayIndexOutOfBound Exception, Thanks
> in
> advance.
>
> --
> http://daily.appspot.com/food/
>