You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@crunch.apache.org by Quentin Ambard <qu...@gmail.com> on 2013/05/28 21:02:04 UTC
example with complex object as emitter
Hi,
I'd like to know if there is an example of a paralleleDo that emit a string
has key, but a more complex object as value, for example the following
class :
StringAndInteger {
String myString;
Integer myInteger;
}
If I'm not wrong the signature of the function should be something like
this (I'm reading the values from hbase):
parallelDo(String name, DoFn<Pair<ImmutableBytesWritable, Result>,
Pair<String, StringAndInteger>> doFn, WritableTableType<String,
StringAndInteger> type);
I have to say I'm a bit lost with the WritableTypes of the parallelDo
PTableType, and a nice example would be very welcome !
Thanks
--
Quentin Ambard
Re: example with complex object as emitter
Posted by Quentin Ambard <qu...@gmail.com>.
Great ! I'm writing the same kind of job, very helpfull.
Thanks
2013/5/28 Josh Wills <jw...@cloudera.com>
> The Avro-based PTypeFamily has support for reflection-based serialization
> of simple types. There's an example in the testAvroReflects() test method
> in the page rank integration test:
>
>
> https://github.com/apache/crunch/blob/master/crunch-core/src/it/java/org/apache/crunch/PageRankIT.java
>
> There are some limits on what you can serialize via Avro reflection (i.e.,
> you need to keep the fields of the class to primitives and simple
> array/collection types), and you need to be sure to include a no-arg
> constructor for the class.
>
>
> On Tue, May 28, 2013 at 12:02 PM, Quentin Ambard <quentin.ambard@gmail.com
> > wrote:
>
>> Hi,
>> I'd like to know if there is an example of a paralleleDo that emit a
>> string has key, but a more complex object as value, for example the
>> following class :
>>
>> StringAndInteger {
>> String myString;
>> Integer myInteger;
>> }
>>
>> If I'm not wrong the signature of the function should be something like
>> this (I'm reading the values from hbase):
>>
>> parallelDo(String name, DoFn<Pair<ImmutableBytesWritable,
>> Result>, Pair<String, StringAndInteger>> doFn, WritableTableType<String,
>> StringAndInteger> type);
>>
>> I have to say I'm a bit lost with the WritableTypes of the parallelDo
>> PTableType, and a nice example would be very welcome !
>>
>> Thanks
>>
>>
>> --
>> Quentin Ambard
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>
--
Quentin Ambard
Re: example with complex object as emitter
Posted by Josh Wills <jw...@cloudera.com>.
The Avro-based PTypeFamily has support for reflection-based serialization
of simple types. There's an example in the testAvroReflects() test method
in the page rank integration test:
https://github.com/apache/crunch/blob/master/crunch-core/src/it/java/org/apache/crunch/PageRankIT.java
There are some limits on what you can serialize via Avro reflection (i.e.,
you need to keep the fields of the class to primitives and simple
array/collection types), and you need to be sure to include a no-arg
constructor for the class.
On Tue, May 28, 2013 at 12:02 PM, Quentin Ambard
<qu...@gmail.com>wrote:
> Hi,
> I'd like to know if there is an example of a paralleleDo that emit a
> string has key, but a more complex object as value, for example the
> following class :
>
> StringAndInteger {
> String myString;
> Integer myInteger;
> }
>
> If I'm not wrong the signature of the function should be something like
> this (I'm reading the values from hbase):
>
> parallelDo(String name, DoFn<Pair<ImmutableBytesWritable, Result>,
> Pair<String, StringAndInteger>> doFn, WritableTableType<String,
> StringAndInteger> type);
>
> I have to say I'm a bit lost with the WritableTypes of the parallelDo
> PTableType, and a nice example would be very welcome !
>
> Thanks
>
>
> --
> Quentin Ambard
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>