Can I use MapWritable as a key?

[Choonho Son <ch...@gmail.com>]

I am newbie.

Most of example shows that,
job.setOutputKeyClass(Text.class);

is it possible job.setOutputKeyClass(MapWritable.class);

because my key is combination of values(src IP, src Port, dst Port,
dst IP), so I want to use MapWritable as a key.

example code is like:

MapWritable mkey = new MapWritable();
MapWritable mval = new MapWritable();

mkey.put(new Text("sip"), new Text(array[3]));
mkey.put(new Text("sport"), new Text(array[5]));
mkey.put(new Text("proto"), new Text(array[7]));
mkey.put(new Text("dport"), new Text(array[6]));
mkey.put(new Text("dip"), new Text(array[4]));


mval.put(new Text("bcount"), new IntWritable(bcount));
mval.put(new Text("pcount"), new IntWritable(pcount));

context.write(mkey, mval);

Re: Can I use MapWritable as a key?

[John Armstrong <jo...@ccri.com>]

On Tue, 19 Jul 2011 17:02:32 -0700, Choonho Son <ch...@gmail.com>
wrote:
> is it possible job.setOutputKeyClass(MapWritable.class);

As others have said, MapWritable doesn't implement Comparable, so it can't
be used as a key.  The ArrayWritable of Texts is one idea, but I'd suggest
instead implementing your OWN WritableComparable class MyKey, which would
contain the four strings as member fields.

Just make a Plain Old Java Object, implement Writable by telling it how to
write and read the fields one at a time (and in the same order!), and
implement comparable as usual for POJOs (remember to override .equals() and
.hashCode() too!).  This way you get type-safety as well, and you know
exactly what your keys will look like.

Re: Can I use MapWritable as a key?

[Harsh J <ha...@cloudera.com>]

Btw, also checkout Avro's MapReduce components. Its a much better
serialization framework, and you'll have lesser issues figuring out
datatypes to use + more performance from good use of codecs.

On Wed, Jul 20, 2011 at 11:37 AM, Harsh J <ha...@cloudera.com> wrote:
> If your key is a "fixed" one of four attributes, why not simply use an
> ArrayWritable of Text objects, over a MapWritable?
>
> On Wed, Jul 20, 2011 at 5:32 AM, Choonho Son <ch...@gmail.com> wrote:
>> I am newbie.
>>
>> Most of example shows that,
>> job.setOutputKeyClass(Text.class);
>>
>> is it possible job.setOutputKeyClass(MapWritable.class);
>>
>> because my key is combination of values(src IP, src Port, dst Port,
>> dst IP), so I want to use MapWritable as a key.
>>
>> example code is like:
>>
>> MapWritable mkey = new MapWritable();
>> MapWritable mval = new MapWritable();
>>
>> mkey.put(new Text("sip"), new Text(array[3]));
>> mkey.put(new Text("sport"), new Text(array[5]));
>> mkey.put(new Text("proto"), new Text(array[7]));
>> mkey.put(new Text("dport"), new Text(array[6]));
>> mkey.put(new Text("dip"), new Text(array[4]));
>>
>>
>> mval.put(new Text("bcount"), new IntWritable(bcount));
>> mval.put(new Text("pcount"), new IntWritable(pcount));
>>
>> context.write(mkey, mval);
>>
>
>
>
> --
> Harsh J
>



-- 
Harsh J

Re: Can I use MapWritable as a key?

[Harsh J <ha...@cloudera.com>]

If your key is a "fixed" one of four attributes, why not simply use an
ArrayWritable of Text objects, over a MapWritable?

On Wed, Jul 20, 2011 at 5:32 AM, Choonho Son <ch...@gmail.com> wrote:
> I am newbie.
>
> Most of example shows that,
> job.setOutputKeyClass(Text.class);
>
> is it possible job.setOutputKeyClass(MapWritable.class);
>
> because my key is combination of values(src IP, src Port, dst Port,
> dst IP), so I want to use MapWritable as a key.
>
> example code is like:
>
> MapWritable mkey = new MapWritable();
> MapWritable mval = new MapWritable();
>
> mkey.put(new Text("sip"), new Text(array[3]));
> mkey.put(new Text("sport"), new Text(array[5]));
> mkey.put(new Text("proto"), new Text(array[7]));
> mkey.put(new Text("dport"), new Text(array[6]));
> mkey.put(new Text("dip"), new Text(array[4]));
>
>
> mval.put(new Text("bcount"), new IntWritable(bcount));
> mval.put(new Text("pcount"), new IntWritable(pcount));
>
> context.write(mkey, mval);
>



-- 
Harsh J

Re: Can I use MapWritable as a key?

[rajesh putta <ra...@gmail.com>]

Hi,
As per my knowledege is concerned
MapWritable doesn't implement Comparable, so I think it cannot be used as a
key. If u want that functionality, then u have to have a subclass that
implements Comparable and there u will define your key comparison logic.or
the other option would be to use SortedMapWritable as key.


Thanks& Regards
Rajesh Putta
M Tech CSE
IIIT-H

On Wed, Jul 20, 2011 at 5:32 AM, Choonho Son <ch...@gmail.com> wrote:

> I am newbie.
>
> Most of example shows that,
> job.setOutputKeyClass(Text.class);
>
> is it possible job.setOutputKeyClass(MapWritable.class);
>
> because my key is combination of values(src IP, src Port, dst Port,
> dst IP), so I want to use MapWritable as a key.
>
> example code is like:
>
> MapWritable mkey = new MapWritable();
> MapWritable mval = new MapWritable();
>
> mkey.put(new Text("sip"), new Text(array[3]));
> mkey.put(new Text("sport"), new Text(array[5]));
> mkey.put(new Text("proto"), new Text(array[7]));
> mkey.put(new Text("dport"), new Text(array[6]));
> mkey.put(new Text("dip"), new Text(array[4]));
>
>
> mval.put(new Text("bcount"), new IntWritable(bcount));
> mval.put(new Text("pcount"), new IntWritable(pcount));
>
> context.write(mkey, mval);
>