You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Check Peck <co...@gmail.com> on 2015/01/13 00:42:33 UTC
How to change a particular field value after decoding the original
byte array?
I have an Avro Schema which is like this -
{
"type":"record",
"name":"new_user",
"namespace":"com.hello",
"fields":[
{
"name":"user_id",
"type":[
"long",
"null"
]
},
{
"name":"segment",
"type":[
"string",
"null"
]
}
]
}
I am using my above Avro Schema like this to serialize the data and which
gives me a Byte Array and works fine -
public static void main(String[] args) throws IOException {
Schema schema = new Parser()
.parse("{ \"type\":\"record\", \"name\":\"new_user\",
\"namespace\":\"com.hello\", \"fields\":[ { \"name\":\"user_id\",
\"type\":[ \"long\", \"null\" ] }, { \"name\":\"segment\", \"type\":[
\"string\", \"null\" ] } ] }");
byte[] originalAvrodata = getAvroBinaryData(schema);
// how to get newAvroData byte array in which user_id
// is change to some other random long number?
}
private static byte[] getAvroBinaryData(Schema schema) throws
IOException {
GenericRecord record = new GenericData.Record(schema);
record.put("user_id", 123456L);
record.put("segment", "hello");
GenericDatumWriter<GenericRecord> writer = new
GenericDatumWriter<GenericRecord>(schema);
ByteArrayOutputStream os = new ByteArrayOutputStream();
Encoder e = EncoderFactory.get().binaryEncoder(os, null);
writer.write(record, e);
e.flush();
byte[] byteData = os.toByteArray();
return byteData;
}
I need to decode the `originalAvrodata` byte array and then change the
`user_id` field value to some other `long` number and then construct a
`newAvroData` byte array using the same schema which should have `user_id`
field value to some random `long` number. Is this possible to do by any
chance using Avro?
Re: How to change a particular field value after decoding the
original byte array?
Posted by Joey Echeverria <jo...@cloudera.com>.
You can re-use the GenericRecord object you used before and then just
re-serialize. If you've been given the byte array, then you can
deserialize to a GenericRecord using something similar to this:
reader = new GenericDatumReader<GenericRecord>(schema);
Decoder decoder = DecoderFactory.get().binaryDecoder(bytes, null);
GenericRecord record = reader.read(null, decoder);
record.put("user_id", 123456L);
-Joey
On Mon, Jan 12, 2015 at 5:21 PM, Check Peck <co...@gmail.com> wrote:
> Yes that's what I am looking for. Question is - How do I update the
> GenericRecord, I want to update user_id field with some new random long
> number.?
>
> Can you provide an example how can I update GenericRecord and then serialize
> it to a new Byte Array?
>
> On Mon, Jan 12, 2015 at 4:34 PM, Joey Echeverria <jo...@cloudera.com> wrote:
>>
>> I'm not sure I understand the question. You can certainly deserialize
>> the byte array back into a GenericRecord. You can then update that
>> GenericRecord and serialize that to a new byte array.
>>
>> Is that what you're looking for?
>>
>> -Joey
>>
>> On Mon, Jan 12, 2015 at 3:42 PM, Check Peck <co...@gmail.com>
>> wrote:
>> > I have an Avro Schema which is like this -
>> >
>> > {
>> > "type":"record",
>> > "name":"new_user",
>> > "namespace":"com.hello",
>> > "fields":[
>> > {
>> > "name":"user_id",
>> > "type":[
>> > "long",
>> > "null"
>> > ]
>> > },
>> > {
>> > "name":"segment",
>> > "type":[
>> > "string",
>> > "null"
>> > ]
>> > }
>> > ]
>> > }
>> >
>> > I am using my above Avro Schema like this to serialize the data and
>> > which
>> > gives me a Byte Array and works fine -
>> >
>> > public static void main(String[] args) throws IOException {
>> > Schema schema = new Parser()
>> > .parse("{ \"type\":\"record\", \"name\":\"new_user\",
>> > \"namespace\":\"com.hello\", \"fields\":[ { \"name\":\"user_id\",
>> > \"type\":[
>> > \"long\", \"null\" ] }, { \"name\":\"segment\", \"type\":[ \"string\",
>> > \"null\" ] } ] }");
>> >
>> > byte[] originalAvrodata = getAvroBinaryData(schema);
>> >
>> > // how to get newAvroData byte array in which user_id
>> > // is change to some other random long number?
>> > }
>> >
>> > private static byte[] getAvroBinaryData(Schema schema) throws
>> > IOException {
>> > GenericRecord record = new GenericData.Record(schema);
>> > record.put("user_id", 123456L);
>> > record.put("segment", "hello");
>> >
>> > GenericDatumWriter<GenericRecord> writer = new
>> > GenericDatumWriter<GenericRecord>(schema);
>> > ByteArrayOutputStream os = new ByteArrayOutputStream();
>> >
>> > Encoder e = EncoderFactory.get().binaryEncoder(os, null);
>> >
>> > writer.write(record, e);
>> > e.flush();
>> > byte[] byteData = os.toByteArray();
>> > return byteData;
>> > }
>> >
>> >
>> >
>> > I need to decode the `originalAvrodata` byte array and then change the
>> > `user_id` field value to some other `long` number and then construct a
>> > `newAvroData` byte array using the same schema which should have
>> > `user_id`
>> > field value to some random `long` number. Is this possible to do by any
>> > chance using Avro?
>>
>>
>>
>> --
>> Joey Echeverria
>
>
--
Joey Echeverria
Re: How to change a particular field value after decoding the
original byte array?
Posted by Check Peck <co...@gmail.com>.
Yes that's what I am looking for. Question is - How do I update the
GenericRecord, I want to update user_id field with some new random long
number.?
Can you provide an example how can I update GenericRecord and then
serialize it to a new Byte Array?
On Mon, Jan 12, 2015 at 4:34 PM, Joey Echeverria <jo...@cloudera.com> wrote:
> I'm not sure I understand the question. You can certainly deserialize
> the byte array back into a GenericRecord. You can then update that
> GenericRecord and serialize that to a new byte array.
>
> Is that what you're looking for?
>
> -Joey
>
> On Mon, Jan 12, 2015 at 3:42 PM, Check Peck <co...@gmail.com>
> wrote:
> > I have an Avro Schema which is like this -
> >
> > {
> > "type":"record",
> > "name":"new_user",
> > "namespace":"com.hello",
> > "fields":[
> > {
> > "name":"user_id",
> > "type":[
> > "long",
> > "null"
> > ]
> > },
> > {
> > "name":"segment",
> > "type":[
> > "string",
> > "null"
> > ]
> > }
> > ]
> > }
> >
> > I am using my above Avro Schema like this to serialize the data and which
> > gives me a Byte Array and works fine -
> >
> > public static void main(String[] args) throws IOException {
> > Schema schema = new Parser()
> > .parse("{ \"type\":\"record\", \"name\":\"new_user\",
> > \"namespace\":\"com.hello\", \"fields\":[ { \"name\":\"user_id\",
> \"type\":[
> > \"long\", \"null\" ] }, { \"name\":\"segment\", \"type\":[ \"string\",
> > \"null\" ] } ] }");
> >
> > byte[] originalAvrodata = getAvroBinaryData(schema);
> >
> > // how to get newAvroData byte array in which user_id
> > // is change to some other random long number?
> > }
> >
> > private static byte[] getAvroBinaryData(Schema schema) throws
> > IOException {
> > GenericRecord record = new GenericData.Record(schema);
> > record.put("user_id", 123456L);
> > record.put("segment", "hello");
> >
> > GenericDatumWriter<GenericRecord> writer = new
> > GenericDatumWriter<GenericRecord>(schema);
> > ByteArrayOutputStream os = new ByteArrayOutputStream();
> >
> > Encoder e = EncoderFactory.get().binaryEncoder(os, null);
> >
> > writer.write(record, e);
> > e.flush();
> > byte[] byteData = os.toByteArray();
> > return byteData;
> > }
> >
> >
> >
> > I need to decode the `originalAvrodata` byte array and then change the
> > `user_id` field value to some other `long` number and then construct a
> > `newAvroData` byte array using the same schema which should have
> `user_id`
> > field value to some random `long` number. Is this possible to do by any
> > chance using Avro?
>
>
>
> --
> Joey Echeverria
>
Re: How to change a particular field value after decoding the
original byte array?
Posted by Joey Echeverria <jo...@cloudera.com>.
I'm not sure I understand the question. You can certainly deserialize
the byte array back into a GenericRecord. You can then update that
GenericRecord and serialize that to a new byte array.
Is that what you're looking for?
-Joey
On Mon, Jan 12, 2015 at 3:42 PM, Check Peck <co...@gmail.com> wrote:
> I have an Avro Schema which is like this -
>
> {
> "type":"record",
> "name":"new_user",
> "namespace":"com.hello",
> "fields":[
> {
> "name":"user_id",
> "type":[
> "long",
> "null"
> ]
> },
> {
> "name":"segment",
> "type":[
> "string",
> "null"
> ]
> }
> ]
> }
>
> I am using my above Avro Schema like this to serialize the data and which
> gives me a Byte Array and works fine -
>
> public static void main(String[] args) throws IOException {
> Schema schema = new Parser()
> .parse("{ \"type\":\"record\", \"name\":\"new_user\",
> \"namespace\":\"com.hello\", \"fields\":[ { \"name\":\"user_id\", \"type\":[
> \"long\", \"null\" ] }, { \"name\":\"segment\", \"type\":[ \"string\",
> \"null\" ] } ] }");
>
> byte[] originalAvrodata = getAvroBinaryData(schema);
>
> // how to get newAvroData byte array in which user_id
> // is change to some other random long number?
> }
>
> private static byte[] getAvroBinaryData(Schema schema) throws
> IOException {
> GenericRecord record = new GenericData.Record(schema);
> record.put("user_id", 123456L);
> record.put("segment", "hello");
>
> GenericDatumWriter<GenericRecord> writer = new
> GenericDatumWriter<GenericRecord>(schema);
> ByteArrayOutputStream os = new ByteArrayOutputStream();
>
> Encoder e = EncoderFactory.get().binaryEncoder(os, null);
>
> writer.write(record, e);
> e.flush();
> byte[] byteData = os.toByteArray();
> return byteData;
> }
>
>
>
> I need to decode the `originalAvrodata` byte array and then change the
> `user_id` field value to some other `long` number and then construct a
> `newAvroData` byte array using the same schema which should have `user_id`
> field value to some random `long` number. Is this possible to do by any
> chance using Avro?
--
Joey Echeverria