You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Check Peck <co...@gmail.com> on 2014/02/24 03:08:42 UTC
How to Avro Binary encode the JSON String using Apache Avro?
I am trying to avro binary encode my JSON String. Below is my JSON String
and I have created a simple method which will do the conversion but I am
not sure whether the way I am doing is correct or not?
public static void main(String args[]) throws Exception{
try{
Schema schema = new
Parser().parse((TestExample.class.getResourceAsStream("/3233.avsc")));
String json="{"+
" \"location\" : {"+
" \"devices\":["+
" {"+
" \"did\":\"9abd09-439bcd-629a8f\","+
" \"dt\":\"browser\","+
" \"usl\":{"+
" \"pos\":{"+
" \"source\":\"GPS\","+
" \"lat\":90.0,"+
" \"long\":101.0,"+
" \"acc\":100"+
" },"+
" \"addSource\":\"LL\","+
" \"add\":["+
" {"+
" \"val\":\"2123\","+
" \"type\" : \"NUM\""+
" },"+
" {"+
" \"val\":\"Harris ST\","+
" \"type\" : \"ST\""+
" }"+
" ],"+
" \"ei\":{"+
" \"ibm\":true,"+
" \"sr\":10,"+
" \"ienz\":true,"+
" \"enz\":100,"+
" \"enr\":10"+
" },"+
" \"lm\":1390598086120"+
" }"+
" }"+
" ],"+
" \"ver\" : \"1.0\""+
" }"+
"}";
byte[] avroByteArray = fromJsonToAvro(json,schema);
} catch (Exception ex) {
// log an exception
}
Below method will convert my JSON String to Avro Binary encoded -
private static byte[] fromJsonToAvro(String json, Schema schema) throws
Exception {
InputStream input = new ByteArrayInputStream(json.getBytes());
DataInputStream din = new DataInputStream(input);
Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
DatumReader<Object> reader = new GenericDatumReader<Object>(schema);
Object datum = reader.read(null, decoder);
GenericDatumWriter<Object> w = new
GenericDatumWriter<Object>(schema);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
Encoder e = EncoderFactory.get().binaryEncoder(outputStream, null);
w.write(datum, e);
e.flush();
return outputStream.toByteArray();
}
Can anyone take a look and let me know whether the way I am trying to avro
binary my JSON String is correct or not?
Re: How to Avro Binary encode the JSON String using Apache Avro?
Posted by Gary Steelman <ga...@gmail.com>.
Hey Check,
I asked a very similar question a few days ago and Dave McAlpin (
dmcalpin@inome.com) was able to help with. He sent out a couple
general-purpose serialization and deserialization functions for JSON to
String and back to JSON, but this can be adapted to JSON->byte[]->JSON
easily:
Here are some utility functions we've used for serialization to and from
JSON. Something similar should work for binary.
public <T> String avroEncodeAsJson(Class<T> clazz, Object object) {
String avroEncodedJson = null;
try {
if (object == null || !(object instanceof SpecificRecord)) {
return null;
}
T record = (T) object;
Schema schema = ((SpecificRecord) record).getSchema();
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder e = EncoderFactory.get().jsonEncoder(schema, out);
SpecificDatumWriter<T> w = new SpecificDatumWriter<T>(clazz);
w.write(record, e);
e.flush();
avroEncodedJson = new String(out.toByteArray());
} catch (IOException e) {
e.printStackTrace();
}
return avroEncodedJson;
}
public <T> T jsonDecodeToAvro(String inputString, Class<T> className,
Schema schema) {
T returnObject = null;
try {
JsonDecoder jsonDecoder = DecoderFactory.get().jsonDecoder(schema,
inputString);
SpecificDatumReader<T> reader = new
SpecificDatumReader<T>(className);
returnObject = reader.read(null, jsonDecoder);
} catch (IOException e) {
e.printStackTrace();
}
return returnObject;
}
Hope that helps,
Gary
On Mon, Feb 24, 2014 at 9:54 AM, Check Peck <co...@gmail.com> wrote:
>
> On Mon, Feb 24, 2014 at 7:43 AM, Adrian Hains <ah...@gmail.com> wrote:
>
>> org.apache.avro.tool.DataFileWriteTool
>>
>
>
> I just read it and the description says "Reads new-line delimited JSON
> records and writers an Avro data file."
>
> And I need to Avro Binary Encode my JSON String into ByteArray and I guess
> they both are different correct?
>
Re: How to Avro Binary encode the JSON String using Apache Avro?
Posted by Check Peck <co...@gmail.com>.
On Mon, Feb 24, 2014 at 7:43 AM, Adrian Hains <ah...@gmail.com> wrote:
> org.apache.avro.tool.DataFileWriteTool
>
I just read it and the description says "Reads new-line delimited JSON
records and writers an Avro data file."
And I need to Avro Binary Encode my JSON String into ByteArray and I guess
they both are different correct?
Re: How to Avro Binary encode the JSON String using Apache Avro?
Posted by Adrian Hains <ah...@gmail.com>.
You can take a look at the org.apache.avro.tool.DataFileWriteTool class in
the avro tools source. It looks similar to what you have, but I only
glanced at it.
-a
On Sun, Feb 23, 2014 at 6:08 PM, Check Peck <co...@gmail.com> wrote:
> I am trying to avro binary encode my JSON String. Below is my JSON String
> and I have created a simple method which will do the conversion but I am
> not sure whether the way I am doing is correct or not?
>
> public static void main(String args[]) throws Exception{
> try{
> Schema schema = new
> Parser().parse((TestExample.class.getResourceAsStream("/3233.avsc")));
> String json="{"+
> " \"location\" : {"+
> " \"devices\":["+
> " {"+
> " \"did\":\"9abd09-439bcd-629a8f\","+
> " \"dt\":\"browser\","+
> " \"usl\":{"+
> " \"pos\":{"+
> " \"source\":\"GPS\","+
> " \"lat\":90.0,"+
> " \"long\":101.0,"+
> " \"acc\":100"+
> " },"+
> " \"addSource\":\"LL\","+
> " \"add\":["+
> " {"+
> " \"val\":\"2123\","+
> " \"type\" : \"NUM\""+
> " },"+
> " {"+
> " \"val\":\"Harris ST\","+
> " \"type\" : \"ST\""+
> " }"+
> " ],"+
> " \"ei\":{"+
> " \"ibm\":true,"+
> " \"sr\":10,"+
> " \"ienz\":true,"+
> " \"enz\":100,"+
> " \"enr\":10"+
> " },"+
> " \"lm\":1390598086120"+
> " }"+
> " }"+
> " ],"+
> " \"ver\" : \"1.0\""+
> " }"+
> "}";
>
> byte[] avroByteArray = fromJsonToAvro(json,schema);
>
> } catch (Exception ex) {
> // log an exception
> }
>
>
> Below method will convert my JSON String to Avro Binary encoded -
>
> private static byte[] fromJsonToAvro(String json, Schema schema)
> throws Exception {
>
> InputStream input = new ByteArrayInputStream(json.getBytes());
> DataInputStream din = new DataInputStream(input);
>
> Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
>
> DatumReader<Object> reader = new
> GenericDatumReader<Object>(schema);
> Object datum = reader.read(null, decoder);
>
>
> GenericDatumWriter<Object> w = new
> GenericDatumWriter<Object>(schema);
> ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
>
> Encoder e = EncoderFactory.get().binaryEncoder(outputStream, null);
>
> w.write(datum, e);
> e.flush();
>
> return outputStream.toByteArray();
> }
>
>
> Can anyone take a look and let me know whether the way I am trying to avro
> binary my JSON String is correct or not?
>