Multiple Output Value Classes

[Stu Hood <st...@mailtrust.com>]

Hey gang,

I know that map/reduce functions will accept any subclass of Writable as input values if the Mapper/Reducer classes are declared as <WritableComparable, Writable, ...>. Then you can use instanceOf to determine the type you received.

But I'm trying to _output_ multiple different value classes from a Mapper, and not having any luck. If I declare the Mapper as <..., WritableComparable, Writable> for instance, it refuses to accept anything other that a Writable as output, and throws a "Type mismatch in value from map: expected org.apache.hadoop.io.Writable, received org.apache.hadoop.io.Text", even though Writable is only an interface, which Text obviously implements.

Any ideas? At this point, any loophole that would allow me to output 2 arbitrary subclasses of Writable simultaneously would be great.

Thanks,

Stu Hood
Architecture Software Developer
Mailtrust, a Division of Rackspace

Re: Multiple Output Value Classes

[Doug Cutting <cu...@apache.org>]

Stu Hood wrote:
> But I'm trying to _output_ multiple different value classes from a Mapper, and not having any luck.

You can wrap things in ObjectWritable.  When writing, this records the 
class name with each instance, then, when reading, constructs an 
appropriate instance and reads it.  It can wrap Writable, String, 
primitive types, and arrays of these.

Doug

Re: Multiple Output Value Classes

[Riccardo Boscolo <dr...@netseer.com>]

Stu,

you can extend this class and use it as map output value:

http://hadoop.apache.org/core/docs/r0.15.3/api/org/apache/hadoop/io/GenericWritable.html

RB

On 3/16/08, Stu Hood <st...@mailtrust.com> wrote:
>
> Hey gang,
>
> I know that map/reduce functions will accept any subclass of Writable as
> input values if the Mapper/Reducer classes are declared as
> <WritableComparable, Writable, ...>. Then you can use instanceOf to
> determine the type you received.
>
> But I'm trying to _output_ multiple different value classes from a Mapper,
> and not having any luck. If I declare the Mapper as <...,
> WritableComparable, Writable> for instance, it refuses to accept anything
> other that a Writable as output, and throws a "Type mismatch in value from
> map: expected org.apache.hadoop.io.Writable, received
> org.apache.hadoop.io.Text", even though Writable is only an interface,
> which Text obviously implements.
>
> Any ideas? At this point, any loophole that would allow me to output 2
> arbitrary subclasses of Writable simultaneously would be great.
>
> Thanks,
>
>
> Stu Hood
> Architecture Software Developer
> Mailtrust, a Division of Rackspace
>
>


-- 
-------------------------------
Riccardo Boscolo, PhD
V.P. of Core Technology
Netseer Inc.
11943 Montana Ave, Suite 200
Los Angeles, CA 90049
T: 310-597-4482
F: 310-597-4489
Email: drboscolo@netseer.com
-------------------------------