Problem running an Non Avro to Avro Job

[Rohini U <ro...@gmail.com>]

Hi All,

I am using avro 15.3 and CDH3. I have a query in running a Non Avro to Avro
Job. My mapper is non Avro mapper and it outputs a avro data.
When I run this, I get an error saying not a Pair Schema. Does the key
always have to be a Pair Schema?

public void map(Text key, Text value,
                OutputCollector<AvroWrapper<TestPair>, NullWritable> output,
                Reporter reporter) throws IOException {
            String valueString = value.toString();
            String[] toks = valueString.split("\t");
            TestPair p = new TestPair();
            p.left = toks[0];
            p.right = toks[1];


            output.collect(new AvroWrapper<TestPair>(p),wt);
        }

 Thanks,
-Rohini

Re: Problem running an Non Avro to Avro Job

[Chris Wilkes <cw...@gmail.com>]

In  org.apache.avro.mapred.HadoopMapper:

public void collect(OUT datum) throws IOException {
      if (isMapOnly) {
        // don't require a Pair for datum
      } else {
        // cast datum to a pair
     }
 }

so in your case you probably just want to set the number of reducers
to 0 to create an avro record without a Pair as the output value (you
mention "key" below btw).

Actually re-reading it looks like you are trying to make a Pair
object.  Why are you putting that inside an AvroWrapper?  Just do
  collector.collect(p)
And set your map output schema to TestPair.SCHEMA$ where SCHEMA$ =
Pair.getPairSchema(keySChema, valueSchema)

Chris

On Thu, Sep 15, 2011 at 6:10 AM, Rohini U <ro...@gmail.com> wrote:
> Hi All,
>
> I am using avro 15.3 and CDH3. I have a query in running a Non Avro to Avro
> Job. My mapper is non Avro mapper and it outputs a avro data.
> When I run this, I get an error saying not a Pair Schema. Does the key
> always have to be a Pair Schema?
>
> public void map(Text key, Text value,
>                 OutputCollector<AvroWrapper<TestPair>, NullWritable> output,
>                 Reporter reporter) throws IOException {
>             String valueString = value.toString();
>             String[] toks = valueString.split("\t");
>             TestPair p = new TestPair();
>             p.left = toks[0];
>             p.right = toks[1];
>
>
>             output.collect(new AvroWrapper<TestPair>(p),wt);
>         }
>
>  Thanks,
> -Rohini
>