You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mahesh Balija <ba...@gmail.com> on 2013/04/30 08:33:16 UTC
Re: reducer gets values with empty attributes
Hi Alex,
Can you please attach your code? and the sample input data.
Best,
Mahesh Balija,
Calsoft Labs.
On Tue, Apr 30, 2013 at 2:29 AM, <al...@aim.com> wrote:
>
> Hello,
>
> I try to write mapreduce program in hadoop -1.0.4. using mapred libs. I have a map function which gets
>
> keys and creates a different object with a few attributes like id and etc and passes it to reducer function using
>
>
> output.collect(key, value);
>
> Reducer gets keys, but values has empty fields (like id and etc) , although map correctly assigns these fields to each value.
>
> Any ideas why this happens?
>
> Thanks.
> Alex.
>
>
Re: reducer gets values with empty attributes
Posted by al...@aim.com.
Hi,
Here is the map and reduce part of the code
public void map(Text key, Writable value, OutputCollector<Text, Document> output, Reporter reporter) throws IOException {
ParseData parseData = (ParseData) value;
Metadata parseMeta = parseData.getParseMeta();
if(parseMeta.getValues("id").length>0)
{
String[] id = parseMeta.getValues("id");
String [] ids=id[0].split(",");
int len=ids.length;
for(int i=0;i<len;i++)
{
Document doc = new Document();
doc.add("id", ids[i]);
output.collect(new Text(ids[i]), doc);
// for testing
List list=doc.getField("id").getValues();
for(int j=0;j<list.size();j++){
System.out.println("map:sending to reducer id field for key "+ids[i]+" as "+list.get(j));
}
//
}
}
}
public void reduce(Text key, Iterator<Document> values, OutputCollector<Text, Document> output, Reporter reporter) throws IOException{
Document doc=values.next();
System.out.println("reducer keys are "+key +" and size of doc is "+doc.getField("id").getValues().size() +"\n");
List list=doc.getField("id").getValues();
System.out.println("reducer: list size is "+list.size() +"\n");
for(int j=0;j<list.size();j++){
System.out.println("reduce:id field for key "+key+" is "+list.get(j));
}
//output.collect(key, doc);
}
System.out.println in map function outputs expected values for id field for each document. However, System.out.println in reducer function outputs keys but no values for id field.
Thanks in advance.
Alex.
-----Original Message-----
From: Mahesh Balija <ba...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Apr 29, 2013 11:33 pm
Subject: Re: reducer gets values with empty attributes
Hi Alex,
Can you please attach your code? and the sample input data.
Best,
Mahesh Balija,
Calsoft Labs.
On Tue, Apr 30, 2013 at 2:29 AM, <al...@aim.com> wrote:
Hello,
I try to write mapreduce program in hadoop -1.0.4. using mapred libs. I have a map function which gets
keys and creates a different object with a few attributes like id and etc and passes it to reducer function using
output.collect(key, value);
Reducer gets keys, but values has empty fields (like id and etc) , although map correctly assigns these fields to each value.
Any ideas why this happens?
Thanks.
Alex.
Re: reducer gets values with empty attributes
Posted by al...@aim.com.
Hi,
Here is the map and reduce part of the code
public void map(Text key, Writable value, OutputCollector<Text, Document> output, Reporter reporter) throws IOException {
ParseData parseData = (ParseData) value;
Metadata parseMeta = parseData.getParseMeta();
if(parseMeta.getValues("id").length>0)
{
String[] id = parseMeta.getValues("id");
String [] ids=id[0].split(",");
int len=ids.length;
for(int i=0;i<len;i++)
{
Document doc = new Document();
doc.add("id", ids[i]);
output.collect(new Text(ids[i]), doc);
// for testing
List list=doc.getField("id").getValues();
for(int j=0;j<list.size();j++){
System.out.println("map:sending to reducer id field for key "+ids[i]+" as "+list.get(j));
}
//
}
}
}
public void reduce(Text key, Iterator<Document> values, OutputCollector<Text, Document> output, Reporter reporter) throws IOException{
Document doc=values.next();
System.out.println("reducer keys are "+key +" and size of doc is "+doc.getField("id").getValues().size() +"\n");
List list=doc.getField("id").getValues();
System.out.println("reducer: list size is "+list.size() +"\n");
for(int j=0;j<list.size();j++){
System.out.println("reduce:id field for key "+key+" is "+list.get(j));
}
//output.collect(key, doc);
}
System.out.println in map function outputs expected values for id field for each document. However, System.out.println in reducer function outputs keys but no values for id field.
Thanks in advance.
Alex.
-----Original Message-----
From: Mahesh Balija <ba...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Apr 29, 2013 11:33 pm
Subject: Re: reducer gets values with empty attributes
Hi Alex,
Can you please attach your code? and the sample input data.
Best,
Mahesh Balija,
Calsoft Labs.
On Tue, Apr 30, 2013 at 2:29 AM, <al...@aim.com> wrote:
Hello,
I try to write mapreduce program in hadoop -1.0.4. using mapred libs. I have a map function which gets
keys and creates a different object with a few attributes like id and etc and passes it to reducer function using
output.collect(key, value);
Reducer gets keys, but values has empty fields (like id and etc) , although map correctly assigns these fields to each value.
Any ideas why this happens?
Thanks.
Alex.
Re: reducer gets values with empty attributes
Posted by al...@aim.com.
Hi,
Here is the map and reduce part of the code
public void map(Text key, Writable value, OutputCollector<Text, Document> output, Reporter reporter) throws IOException {
ParseData parseData = (ParseData) value;
Metadata parseMeta = parseData.getParseMeta();
if(parseMeta.getValues("id").length>0)
{
String[] id = parseMeta.getValues("id");
String [] ids=id[0].split(",");
int len=ids.length;
for(int i=0;i<len;i++)
{
Document doc = new Document();
doc.add("id", ids[i]);
output.collect(new Text(ids[i]), doc);
// for testing
List list=doc.getField("id").getValues();
for(int j=0;j<list.size();j++){
System.out.println("map:sending to reducer id field for key "+ids[i]+" as "+list.get(j));
}
//
}
}
}
public void reduce(Text key, Iterator<Document> values, OutputCollector<Text, Document> output, Reporter reporter) throws IOException{
Document doc=values.next();
System.out.println("reducer keys are "+key +" and size of doc is "+doc.getField("id").getValues().size() +"\n");
List list=doc.getField("id").getValues();
System.out.println("reducer: list size is "+list.size() +"\n");
for(int j=0;j<list.size();j++){
System.out.println("reduce:id field for key "+key+" is "+list.get(j));
}
//output.collect(key, doc);
}
System.out.println in map function outputs expected values for id field for each document. However, System.out.println in reducer function outputs keys but no values for id field.
Thanks in advance.
Alex.
-----Original Message-----
From: Mahesh Balija <ba...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Apr 29, 2013 11:33 pm
Subject: Re: reducer gets values with empty attributes
Hi Alex,
Can you please attach your code? and the sample input data.
Best,
Mahesh Balija,
Calsoft Labs.
On Tue, Apr 30, 2013 at 2:29 AM, <al...@aim.com> wrote:
Hello,
I try to write mapreduce program in hadoop -1.0.4. using mapred libs. I have a map function which gets
keys and creates a different object with a few attributes like id and etc and passes it to reducer function using
output.collect(key, value);
Reducer gets keys, but values has empty fields (like id and etc) , although map correctly assigns these fields to each value.
Any ideas why this happens?
Thanks.
Alex.
Re: reducer gets values with empty attributes
Posted by al...@aim.com.
Hi,
Here is the map and reduce part of the code
public void map(Text key, Writable value, OutputCollector<Text, Document> output, Reporter reporter) throws IOException {
ParseData parseData = (ParseData) value;
Metadata parseMeta = parseData.getParseMeta();
if(parseMeta.getValues("id").length>0)
{
String[] id = parseMeta.getValues("id");
String [] ids=id[0].split(",");
int len=ids.length;
for(int i=0;i<len;i++)
{
Document doc = new Document();
doc.add("id", ids[i]);
output.collect(new Text(ids[i]), doc);
// for testing
List list=doc.getField("id").getValues();
for(int j=0;j<list.size();j++){
System.out.println("map:sending to reducer id field for key "+ids[i]+" as "+list.get(j));
}
//
}
}
}
public void reduce(Text key, Iterator<Document> values, OutputCollector<Text, Document> output, Reporter reporter) throws IOException{
Document doc=values.next();
System.out.println("reducer keys are "+key +" and size of doc is "+doc.getField("id").getValues().size() +"\n");
List list=doc.getField("id").getValues();
System.out.println("reducer: list size is "+list.size() +"\n");
for(int j=0;j<list.size();j++){
System.out.println("reduce:id field for key "+key+" is "+list.get(j));
}
//output.collect(key, doc);
}
System.out.println in map function outputs expected values for id field for each document. However, System.out.println in reducer function outputs keys but no values for id field.
Thanks in advance.
Alex.
-----Original Message-----
From: Mahesh Balija <ba...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Apr 29, 2013 11:33 pm
Subject: Re: reducer gets values with empty attributes
Hi Alex,
Can you please attach your code? and the sample input data.
Best,
Mahesh Balija,
Calsoft Labs.
On Tue, Apr 30, 2013 at 2:29 AM, <al...@aim.com> wrote:
Hello,
I try to write mapreduce program in hadoop -1.0.4. using mapred libs. I have a map function which gets
keys and creates a different object with a few attributes like id and etc and passes it to reducer function using
output.collect(key, value);
Reducer gets keys, but values has empty fields (like id and etc) , although map correctly assigns these fields to each value.
Any ideas why this happens?
Thanks.
Alex.