You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Ross Nordeen <rj...@mtu.edu> on 2011/08/12 23:06:08 UTC
Possible to override the context.write() method in ReduceContext?
Using 0.20.2...
Is it possible to override the context.write() method in ReduceContext? I have an entire set of Reducers that I would like to all use a specific function just before every context.write() but I don't want them to worry about that logic, just to have it handled transparently.
For instance:
Iterator<Text> vit = values.iterator();
if (trans2 != null) {
key = (Text) trans2.transform(key);
}
while (vit.hasNext()) {
Text item = vit.next();
if (trans1 != null) {
item = (Text) trans1.transform(item);
}
context.write(key, item);
}
The logic before/after the write is often different, and there may be writes that happen in different cases. I want to move the if->transform functionality out to the write() function so I can just call context.write(a,b) instead of if (trans1 !=....
My end-goal is something like this:
protected void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
Iterator<Text> vit = values.iterator();
while (vit.hasNext())
context.write(key, vit.next());
}
With the write() method override:
public void write(Text key, Text val) {
if (trans1 != null)
val = trans1.transform(val);
if (trans2 != null)
key = trans2.transform(key);
super.write(key,val);
}
or something similar.
Thanks in advance,
-Ross
--
Ross Nordeen
Computer Networking And Systems Administration
Michigan Technological University
http://www.linkedin.com/in/rjnordee
Re: Possible to override the context.write() method in
ReduceContext?
Posted by Ross Nordeen <rj...@mtu.edu>.
Dino,
Ya that was another method we are using now. thanks
-Ross
--
Ross Nordeen
Computer Networking And Systems Administration
Michigan Technological University
http://www.linkedin.com/in/rjnordee
----- Original Message -----
From: "Dino Kečo" <di...@gmail.com>
To: mapreduce-user@hadoop.apache.org, "Ross" <rj...@mtu.edu>
Sent: Saturday, August 13, 2011 1:29:14 AM GMT -08:00 US/Canada Pacific
Subject: Re: Possible to override the context.write() method in ReduceContext?
Hi Ross,
The first thing that I should do is to create one main reducer and create writeToContext method in it. This method should handle your logic with if(...){...} and context.write(). All other reducers would extend this main reducer and call writeToContext instead context.write.
Thanks,
dino
On Fri, Aug 12, 2011 at 11:06 PM, Ross Nordeen < rjnordee@mtu.edu > wrote:
Using 0.20.2...
Is it possible to override the context.write() method in ReduceContext? I have an entire set of Reducers that I would like to all use a specific function just before every context.write() but I don't want them to worry about that logic, just to have it handled transparently.
For instance:
Iterator<Text> vit = values.iterator();
if (trans2 != null) {
key = (Text) trans2.transform(key);
}
while (vit.hasNext()) {
Text item = vit.next();
if (trans1 != null) {
item = (Text) trans1.transform(item);
}
context.write(key, item);
}
The logic before/after the write is often different, and there may be writes that happen in different cases. I want to move the if->transform functionality out to the write() function so I can just call context.write(a,b) instead of if (trans1 !=....
My end-goal is something like this:
protected void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
Iterator<Text> vit = values.iterator();
while (vit.hasNext())
context.write(key, vit.next());
}
With the write() method override:
public void write(Text key, Text val) {
if (trans1 != null)
val = trans1.transform(val);
if (trans2 != null)
key = trans2.transform(key);
super.write(key,val);
}
or something similar.
Thanks in advance,
-Ross
--
Ross Nordeen
Computer Networking And Systems Administration
Michigan Technological University
http://www.linkedin.com/in/rjnordee
Re: Possible to override the context.write() method in ReduceContext?
Posted by Dino Kečo <di...@gmail.com>.
Hi Ross,
The first thing that I should do is to create one main reducer and create
writeToContext method in it. This method should handle your logic with
if(...){...} and context.write(). All other reducers would extend this main
reducer and call writeToContext instead context.write.
Thanks,
dino
On Fri, Aug 12, 2011 at 11:06 PM, Ross Nordeen <rj...@mtu.edu> wrote:
> Using 0.20.2...
>
> Is it possible to override the context.write() method in ReduceContext? I
> have an entire set of Reducers that I would like to all use a specific
> function just before every context.write() but I don't want them to worry
> about that logic, just to have it handled transparently.
>
> For instance:
>
> Iterator<Text> vit = values.iterator();
>
> if (trans2 != null) {
> key = (Text) trans2.transform(key);
> }
>
> while (vit.hasNext()) {
> Text item = vit.next();
> if (trans1 != null) {
> item = (Text) trans1.transform(item);
> }
> context.write(key, item);
> }
>
> The logic before/after the write is often different, and there may be
> writes that happen in different cases. I want to move the if->transform
> functionality out to the write() function so I can just call
> context.write(a,b) instead of if (trans1 !=....
>
> My end-goal is something like this:
>
> protected void reduce(Text key, Iterable<Text> values, Context context)
> throws IOException, InterruptedException {
> Iterator<Text> vit = values.iterator();
>
> while (vit.hasNext())
> context.write(key, vit.next());
> }
>
> With the write() method override:
>
> public void write(Text key, Text val) {
> if (trans1 != null)
> val = trans1.transform(val);
> if (trans2 != null)
> key = trans2.transform(key);
> super.write(key,val);
> }
>
> or something similar.
>
> Thanks in advance,
>
> -Ross
>
> --
> Ross Nordeen
> Computer Networking And Systems Administration
> Michigan Technological University
> http://www.linkedin.com/in/rjnordee
>
>