You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Roger Hoover <ro...@gmail.com> on 2014/02/05 19:23:06 UTC
Direct conversion from Generic Record to Specific Record
Hi,
I'm working with an existing API that hands me a GenericRecord. Is there
an easy way I can covert it into a SpecificRecord. Really I want to get it
into the code-generated object.
Thanks,
Roger
Re: Direct conversion from Generic Record to Specific Record
Posted by Roger Hoover <ro...@gmail.com>.
Thank you, Mika. I'll do this in mean time. Was wondering if there was a
better way...
On Thu, Feb 6, 2014 at 6:10 AM, Mika Ristimaki <mi...@gmail.com>wrote:
> Hi,
>
> If you don't mind the overhead you can make the conversion through the
> avro binary data. So something like this should work (assuming both records
> are using the same schema)
>
> GenericRecord myRecord = ...
> GenericDatumWriter<GenericRecord> writer = new
> GenericDatumWriter<GenericRecord>(MyCustomRecord.getClassSchema());
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
> writer.write(myRecord, encoder);
> encoder.flush();
>
> byte[] avroData = out.toByteArray();
> out.close();
>
> SpecificDatumReader<MyCustomRecord> reader = new
> SpecificDatumReader<MyCustomRecord>(MyCustomRecord.class);
> Decoder decoder = DecoderFactory.get().binaryDecoder(avroData, null);
> MyCustomRecord myCustomRecord = reader.read(null, decoder);
>
> -Mika
>
> On Feb 6, 2014, at 12:59 AM, Roger Hoover <ro...@gmail.com> wrote:
>
> > Hi Christophe,
> >
> > Thanks for your reply. When you say that we could write a generic
> conversion function, do you mean we can write one that works for all
> schemas? That would be great!
> >
> > I'd like an API something like this:
> >
> > MyCustomRecord record = MyCustomRecord.newBuilder(GenericRecord
> record).build()
> >
> > Thanks,
> >
> > Roger
> >
> >
> > On Wed, Feb 5, 2014 at 10:59 AM, Christophe Taton <
> christophe.taton@gmail.com> wrote:
> > Hi Roger,
> > With the change proposed in
> https://issues.apache.org/jira/browse/AVRO-1443, you would be able to
> create a specific record using a generic builder API. That means we could
> write a generic conversion function that creates specific records.
> > Would that work for you?
> > C.
> >
> >
> > On Wed, Feb 5, 2014 at 10:23 AM, Roger Hoover <ro...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm working with an existing API that hands me a GenericRecord. Is
> there an easy way I can covert it into a SpecificRecord. Really I want to
> get it into the code-generated object.
> >
> > Thanks,
> >
> > Roger
> >
> >
> >
> >
>
>
Re: Direct conversion from Generic Record to Specific Record
Posted by Mika Ristimaki <mi...@gmail.com>.
Hi,
If you don't mind the overhead you can make the conversion through the avro binary data. So something like this should work (assuming both records are using the same schema)
GenericRecord myRecord = ...
GenericDatumWriter<GenericRecord> writer = new GenericDatumWriter<GenericRecord>(MyCustomRecord.getClassSchema());
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
writer.write(myRecord, encoder);
encoder.flush();
byte[] avroData = out.toByteArray();
out.close();
SpecificDatumReader<MyCustomRecord> reader = new SpecificDatumReader<MyCustomRecord>(MyCustomRecord.class);
Decoder decoder = DecoderFactory.get().binaryDecoder(avroData, null);
MyCustomRecord myCustomRecord = reader.read(null, decoder);
-Mika
On Feb 6, 2014, at 12:59 AM, Roger Hoover <ro...@gmail.com> wrote:
> Hi Christophe,
>
> Thanks for your reply. When you say that we could write a generic conversion function, do you mean we can write one that works for all schemas? That would be great!
>
> I'd like an API something like this:
>
> MyCustomRecord record = MyCustomRecord.newBuilder(GenericRecord record).build()
>
> Thanks,
>
> Roger
>
>
> On Wed, Feb 5, 2014 at 10:59 AM, Christophe Taton <ch...@gmail.com> wrote:
> Hi Roger,
> With the change proposed in https://issues.apache.org/jira/browse/AVRO-1443, you would be able to create a specific record using a generic builder API. That means we could write a generic conversion function that creates specific records.
> Would that work for you?
> C.
>
>
> On Wed, Feb 5, 2014 at 10:23 AM, Roger Hoover <ro...@gmail.com> wrote:
> Hi,
>
> I'm working with an existing API that hands me a GenericRecord. Is there an easy way I can covert it into a SpecificRecord. Really I want to get it into the code-generated object.
>
> Thanks,
>
> Roger
>
>
>
>
Re: Direct conversion from Generic Record to Specific Record
Posted by Roger Hoover <ro...@gmail.com>.
Great. Thanks, Christophe.
On Thu, Feb 6, 2014 at 8:12 AM, Christophe Taton <christophe.taton@gmail.com
> wrote:
> I believe we could eventually make a generic function that can rewrite any
> record or record builder, whether specific or generic, into another record
> or record builder, whether specific or generic.
>
> Until this happens, Mika's suggestion is probably the best short-term
> solution.
>
> C.
>
>
> On Wed, Feb 5, 2014 at 2:59 PM, Roger Hoover <ro...@gmail.com>wrote:
>
>> Hi Christophe,
>>
>> Thanks for your reply. When you say that we could write a generic
>> conversion function, do you mean we can write one that works for all
>> schemas? That would be great!
>>
>> I'd like an API something like this:
>>
>> MyCustomRecord record = MyCustomRecord.newBuilder(GenericRecord
>> record).build()
>>
>> Thanks,
>>
>> Roger
>>
>>
>> On Wed, Feb 5, 2014 at 10:59 AM, Christophe Taton <
>> christophe.taton@gmail.com> wrote:
>>
>>> Hi Roger,
>>> With the change proposed in
>>> https://issues.apache.org/jira/browse/AVRO-1443, you would be able to
>>> create a specific record using a generic builder API. That means we could
>>> write a generic conversion function that creates specific records.
>>> Would that work for you?
>>> C.
>>>
>>>
>>> On Wed, Feb 5, 2014 at 10:23 AM, Roger Hoover <ro...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm working with an existing API that hands me a GenericRecord. Is
>>>> there an easy way I can covert it into a SpecificRecord. Really I want to
>>>> get it into the code-generated object.
>>>>
>>>> Thanks,
>>>>
>>>> Roger
>>>>
>>>>
>>>>
>>>
>>
>
Re: Direct conversion from Generic Record to Specific Record
Posted by Doug Cutting <cu...@apache.org>.
SpecificData#deepCopy will make this conversion. It currently fails
for enums, but the fix is easy. Here's a patch that makes that fix
and demonstrates a conversion. If this change is of interest, please
file an issue in Jira.
Doug
Index: lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java
===================================================================
--- lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java
(revision 1564561)
+++ lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java
(working copy)
@@ -933,8 +933,7 @@
case DOUBLE:
return (T)new Double((Double)value);
case ENUM:
- // Enums are immutable; shallow copy will suffice
- return value;
+ return (T)createEnum(value.toString(), schema);
case FIXED:
return (T)createFixed(null, ((GenericFixed) value).bytes(), schema);
case FLOAT:
Index: lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificData.java
===================================================================
--- lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificData.java
(revision 1564561)
+++ lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificData.java
(working copy)
@@ -40,6 +40,8 @@
import org.apache.avro.test.MD5;
import org.apache.avro.test.Kind;
+import org.apache.avro.generic.GenericRecord;
+
public class TestSpecificData {
@Test
@@ -95,6 +97,16 @@
new SpecificDatumReader<Object>());
}
+ @Test public void testConvertGenericToSpecific() {
+ GenericRecord generic = new GenericData.Record(TestRecord.SCHEMA$);
+ generic.put("name", "foo");
+ generic.put("kind", new GenericData.EnumSymbol(Kind.SCHEMA$, "BAR"));
+ generic.put("hash", new GenericData.Fixed
+ (MD5.SCHEMA$, new byte[]{0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5}));
+ TestRecord specific =
+ (TestRecord)SpecificData.get().deepCopy(TestRecord.SCHEMA$, generic);
+ }
+
@Test public void testGetClassSchema() throws Exception {
Assert.assertEquals(TestRecord.getClassSchema(), TestRecord.SCHEMA$);
Assert.assertEquals(MD5.getClassSchema(), MD5.SCHEMA$);
On Thu, Feb 6, 2014 at 8:12 AM, Christophe Taton
<ch...@gmail.com> wrote:
> I believe we could eventually make a generic function that can rewrite any
> record or record builder, whether specific or generic, into another record
> or record builder, whether specific or generic.
>
> Until this happens, Mika's suggestion is probably the best short-term
> solution.
>
> C.
>
>
> On Wed, Feb 5, 2014 at 2:59 PM, Roger Hoover <ro...@gmail.com> wrote:
>>
>> Hi Christophe,
>>
>> Thanks for your reply. When you say that we could write a generic
>> conversion function, do you mean we can write one that works for all
>> schemas? That would be great!
>>
>> I'd like an API something like this:
>>
>> MyCustomRecord record = MyCustomRecord.newBuilder(GenericRecord
>> record).build()
>>
>> Thanks,
>>
>> Roger
>>
>>
>> On Wed, Feb 5, 2014 at 10:59 AM, Christophe Taton
>> <ch...@gmail.com> wrote:
>>>
>>> Hi Roger,
>>> With the change proposed in
>>> https://issues.apache.org/jira/browse/AVRO-1443, you would be able to create
>>> a specific record using a generic builder API. That means we could write a
>>> generic conversion function that creates specific records.
>>> Would that work for you?
>>> C.
>>>
>>>
>>> On Wed, Feb 5, 2014 at 10:23 AM, Roger Hoover <ro...@gmail.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm working with an existing API that hands me a GenericRecord. Is
>>>> there an easy way I can covert it into a SpecificRecord. Really I want to
>>>> get it into the code-generated object.
>>>>
>>>> Thanks,
>>>>
>>>> Roger
>>>>
>>>>
>>>
>>
>
Re: Direct conversion from Generic Record to Specific Record
Posted by Christophe Taton <ch...@gmail.com>.
I believe we could eventually make a generic function that can rewrite any
record or record builder, whether specific or generic, into another record
or record builder, whether specific or generic.
Until this happens, Mika's suggestion is probably the best short-term
solution.
C.
On Wed, Feb 5, 2014 at 2:59 PM, Roger Hoover <ro...@gmail.com> wrote:
> Hi Christophe,
>
> Thanks for your reply. When you say that we could write a generic
> conversion function, do you mean we can write one that works for all
> schemas? That would be great!
>
> I'd like an API something like this:
>
> MyCustomRecord record = MyCustomRecord.newBuilder(GenericRecord
> record).build()
>
> Thanks,
>
> Roger
>
>
> On Wed, Feb 5, 2014 at 10:59 AM, Christophe Taton <
> christophe.taton@gmail.com> wrote:
>
>> Hi Roger,
>> With the change proposed in
>> https://issues.apache.org/jira/browse/AVRO-1443, you would be able to
>> create a specific record using a generic builder API. That means we could
>> write a generic conversion function that creates specific records.
>> Would that work for you?
>> C.
>>
>>
>> On Wed, Feb 5, 2014 at 10:23 AM, Roger Hoover <ro...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I'm working with an existing API that hands me a GenericRecord. Is
>>> there an easy way I can covert it into a SpecificRecord. Really I want to
>>> get it into the code-generated object.
>>>
>>> Thanks,
>>>
>>> Roger
>>>
>>>
>>>
>>
>
Re: Direct conversion from Generic Record to Specific Record
Posted by Roger Hoover <ro...@gmail.com>.
Hi Christophe,
Thanks for your reply. When you say that we could write a generic
conversion function, do you mean we can write one that works for all
schemas? That would be great!
I'd like an API something like this:
MyCustomRecord record = MyCustomRecord.newBuilder(GenericRecord
record).build()
Thanks,
Roger
On Wed, Feb 5, 2014 at 10:59 AM, Christophe Taton <
christophe.taton@gmail.com> wrote:
> Hi Roger,
> With the change proposed in
> https://issues.apache.org/jira/browse/AVRO-1443, you would be able to
> create a specific record using a generic builder API. That means we could
> write a generic conversion function that creates specific records.
> Would that work for you?
> C.
>
>
> On Wed, Feb 5, 2014 at 10:23 AM, Roger Hoover <ro...@gmail.com>wrote:
>
>> Hi,
>>
>> I'm working with an existing API that hands me a GenericRecord. Is there
>> an easy way I can covert it into a SpecificRecord. Really I want to get it
>> into the code-generated object.
>>
>> Thanks,
>>
>> Roger
>>
>>
>>
>
Re: Direct conversion from Generic Record to Specific Record
Posted by Christophe Taton <ch...@gmail.com>.
Hi Roger,
With the change proposed in https://issues.apache.org/jira/browse/AVRO-1443,
you would be able to create a specific record using a generic builder API.
That means we could write a generic conversion function that creates
specific records.
Would that work for you?
C.
On Wed, Feb 5, 2014 at 10:23 AM, Roger Hoover <ro...@gmail.com>wrote:
> Hi,
>
> I'm working with an existing API that hands me a GenericRecord. Is there
> an easy way I can covert it into a SpecificRecord. Really I want to get it
> into the code-generated object.
>
> Thanks,
>
> Roger
>
>
>