You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Russell Jurney <ru...@gmail.com> on 2012/02/09 04:11:51 UTC
Generate Maps
Is it possible to generate maps in Pig? Is this type castable in any
context?
--
Russell Jurney
twitter.com/rjurney
russell.jurney@gmail.com
datasyndrome.com
Re: Generate Maps
Posted by Haitao Yao <ya...@gmail.com>.
The code pasted is wrong, sorry for my mistake.
here's the updated code:
@SuppressWarnings("rawtypes")
public class HEDataConverter extends EvalFunc<Map> {
@Override
public Map<String, String> exec(Tuple input) throws IOException {
byte[] mapValue = ((DataByteArray) input.get(0)).get();
return HEFunctions.parsePayload(mapValue);
}
@Override
public Schema outputSchema(Schema inputSchema) {
FieldSchema newFieldSchema = new FieldSchema("payload", DataType.MAP);
return new Schema(newFieldSchema);
}
}
在 2012-2-9，下午3:48， Russell Jurney 写道：
> Thanks, I'll try this tomorrow.
>
> Sent from my iPad
>
> On Feb 8, 2012, at 11:40 PM, Haitao Yao <ya...@gmail.com> wrote:
>
>> you can write a EvalFuc like this:
>> /**
>> * @author haitao
>> *
>> */
>> public class HEDataConverter extends EvalFunc<Tuple> {
>>
>> private TupleFactory tupleFactory = TupleFactory.getInstance();
>>
>> @Override
>> public Tuple exec(Tuple input) throws IOException {
>> byte[] mapValue = (byte[]) input.get(0);
>> List<Object> all = new ArrayList<Object>(1);
>> all.set(1, HEFunctions.parsePayload(mapValue));
>> return tupleFactory.newTuple(all);
>> }
>>
>> @Override
>> public Schema outputSchema(Schema inputSchema) {
>> FieldSchema newFieldSchema = new FieldSchema("payload", DataType.MAP);
>> return new Schema(newFieldSchema);
>> }
>>
>> }
>>
>> and you will get your map like this:
>>
>> grunt> A = load '/tmp/test_file' using PigStorage(',') as (uid:long, payload:bytearray, ts:long, type:int);
>> grunt> B = foreach A generate he.HEDataConverter(payload);
>> grunt> describe B;
>> B: {payload: map[]}
>>
>> hope this is helpful.
>>
>> 在 2012-2-9，上午11:11， Russell Jurney 写道：
>>
>>> Is it possible to generate maps in Pig? Is this type castable in any
>>> context?
>>>
>>> --
>>> Russell Jurney
>>> twitter.com/rjurney
>>> russell.jurney@gmail.com
>>> datasyndrome.com
>>
Re: Generate Maps
Posted by Russell Jurney <ru...@gmail.com>.
Thanks, I'll try this tomorrow.
Sent from my iPad
On Feb 8, 2012, at 11:40 PM, Haitao Yao <ya...@gmail.com> wrote:
> you can write a EvalFuc like this:
> /**
> * @author haitao
> *
> */
> public class HEDataConverter extends EvalFunc<Tuple> {
>
> private TupleFactory tupleFactory = TupleFactory.getInstance();
>
> @Override
> public Tuple exec(Tuple input) throws IOException {
> byte[] mapValue = (byte[]) input.get(0);
> List<Object> all = new ArrayList<Object>(1);
> all.set(1, HEFunctions.parsePayload(mapValue));
> return tupleFactory.newTuple(all);
> }
>
> @Override
> public Schema outputSchema(Schema inputSchema) {
> FieldSchema newFieldSchema = new FieldSchema("payload", DataType.MAP);
> return new Schema(newFieldSchema);
> }
>
> }
>
> and you will get your map like this:
>
> grunt> A = load '/tmp/test_file' using PigStorage(',') as (uid:long, payload:bytearray, ts:long, type:int);
> grunt> B = foreach A generate he.HEDataConverter(payload);
> grunt> describe B;
> B: {payload: map[]}
>
> hope this is helpful.
>
> 在 2012-2-9，上午11:11， Russell Jurney 写道：
>
>> Is it possible to generate maps in Pig? Is this type castable in any
>> context?
>>
>> --
>> Russell Jurney
>> twitter.com/rjurney
>> russell.jurney@gmail.com
>> datasyndrome.com
>
Re: Generate Maps
Posted by Haitao Yao <ya...@gmail.com>.
you can write a EvalFuc like this:
/**
* @author haitao
*
*/
public class HEDataConverter extends EvalFunc<Tuple> {
private TupleFactory tupleFactory = TupleFactory.getInstance();
@Override
public Tuple exec(Tuple input) throws IOException {
byte[] mapValue = (byte[]) input.get(0);
List<Object> all = new ArrayList<Object>(1);
all.set(1, HEFunctions.parsePayload(mapValue));
return tupleFactory.newTuple(all);
}
@Override
public Schema outputSchema(Schema inputSchema) {
FieldSchema newFieldSchema = new FieldSchema("payload", DataType.MAP);
return new Schema(newFieldSchema);
}
}
and you will get your map like this:
grunt> A = load '/tmp/test_file' using PigStorage(',') as (uid:long, payload:bytearray, ts:long, type:int);
grunt> B = foreach A generate he.HEDataConverter(payload);
grunt> describe B;
B: {payload: map[]}
hope this is helpful.
在 2012-2-9，上午11:11， Russell Jurney 写道：
> Is it possible to generate maps in Pig? Is this type castable in any
> context?
>
> --
> Russell Jurney
> twitter.com/rjurney
> russell.jurney@gmail.com
> datasyndrome.com