You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by tl <tl...@rat.io> on 2016/02/23 02:22:31 UTC
Json.ObjectWriter - "Not the Json schema"
Hi,
I want to convert incoming data to Avro and JSON (and later Parquet). Avro conversion is working okay, but JSON conversion throws the following error that I don’t understand:
Exception in thread "main" java.lang.RuntimeException: Not the Json schema: {"type":"record","name":"Torperf","namespace":"converTor.torperf","fields":[{"name":"descriptor_type","type":"string","default":"torperf 1.0"},
[ … omitted for brevity …]
{"name":"circ_id","type":["null","int"],"doc":"metrics-lib/TorperfResult: int getCircId()"},{"name":"used_by","type":["null","int"],"doc":"metrics-lib/TorperfResult: int getUsedBy()"}],"aliases":["torperfResult"]}
at org.apache.avro.data.Json$ObjectWriter.setSchema(Json.java:117)
at converTor.WriterObject.<init>(WriterObject.java:116)
at converTor.TypeWriter.get(TypeWriter.java:31)
at converTor.ConverTor.main(ConverTor.java:249)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
… but that schema is indeed the schema that I want to use.
This is a snippet of my code:
File schemaFile = new File("schema/jsonSchema.avsc");
Schema.Parser parser = new Schema.Parser();
Schema mySchema = parser.parse(schemaFile) ;
Json.ObjectWriter jsonDatumWriter = new Json.ObjectWriter();
jsonDatumWriter.setSchema(mySchema);
OutputStream out = new FileOutputStream(outputFile);
Encoder encoder = EncoderFactory.get().jsonEncoder(mySchema, out);
Can somebody give me a hint?
Thanks,
Thomas
Re: Json.ObjectWriter - "Not the Json schema"
Posted by tl <tl...@rat.io>.
> On 24.02.2016, at 19:08, Prajwal Tuladhar <pr...@infynyxx.com> wrote:
>
> Can you paste your avro IDL schema?
See below. The full source is available at https://github.com/tomlurge/converTor
@namespace("converTor.torperf")
protocol torperf_descriptor {
@aliases(["torperfResult"])
record Torperf {
string descriptor_type = "torperf 1.0";
/** metrics-lib/TorperfResult: String getSource() */
union { null, string } source;
/** metrics-lib/TorperfResult: int getFileSize() */
union { null, int } filesize;
/** metrics-lib/TorperfResult: long getStartMillis() */
union { null, long } start;
/** metrics-lib/TorperfResult: long getSocketMillis() */
union { null, long } socket;
/** metrics-lib/TorperfResult: long getConnectMillis() */
union { null, long } connect;
/** metrics-lib/TorperfResult: long getNegotiateMillis() */
union { null, long } negotiate;
/** metrics-lib/TorperfResult: long getRequestMillis() */
union { null, long } request;
/** metrics-lib/TorperfResult: long getResponseMillis() */
union { null, long } response;
/** metrics-lib/TorperfResult: long getDataRequestMillis() */
union { null, long } datarequest;
/** metrics-lib/TorperfResult: long getDataResponseMillis() */
union { null, long } dataresponse;
/** metrics-lib/TorperfResult: long getDataCompleteMillis() */
union { null, long } datacomplete;
/** metrics-lib/TorperfResult: int getWriteBytes() */
union { null, int } writebytes;
/** metrics-lib/TorperfResult: int getReadBytes() */
union { null, int } readbytes;
/** metrics-lib/TorperfResult: Boolean didTimeout() */
union { null, boolean } didtimeout;
/** metrics-lib/TorperfResult: long SortedMap<Integer, Long> getDataPercentiles().get(10) */
union { null, long } dataperc10;
/** metrics-lib/TorperfResult: long SortedMap<Integer, Long> getDataPercentiles().get(20) */
union { null, long } dataperc20;
/** metrics-lib/TorperfResult: long SortedMap<Integer, Long> getDataPercentiles().get(30) */
union { null, long } dataperc30;
/** metrics-lib/TorperfResult: long SortedMap<Integer, Long> getDataPercentiles().get(40) */
union { null, long } dataperc40;
/** metrics-lib/TorperfResult: long SortedMap<Integer, Long> getDataPercentiles().get(50) */
union { null, long } dataperc50;
/** metrics-lib/TorperfResult: long SortedMap<Integer, Long> getDataPercentiles().get(60) */
union { null, long } dataperc60;
/** metrics-lib/TorperfResult: long SortedMap<Integer, Long> getDataPercentiles().get(70) */
union { null, long } dataperc70;
/** metrics-lib/TorperfResult: long SortedMap<Integer, Long> getDataPercentiles().get(80) */
union { null, long } dataperc80;
/** metrics-lib/TorperfResult: long SortedMap<Integer, Long> getDataPercentiles().get(90) */
union { null, long } dataperc90;
/** metrics-lib/TorperfResult: long getLaunchMillis()() */
union { null, long } launch;
/** metrics-lib/TorperfResult: long getUsedAtMillis() */
union { null, long } used_at;
/** metrics-lib/TorperfResult: List<String> getPath() */
union { null, array<string> } path;
/** metrics-lib/TorperfResult: List<Long> getBuildTimes() */
union { null, array<long> } buildtimes;
/** metrics-lib/TorperfResult: long getTimeout() */
union { null, long } timeout;
/** metrics-lib/TorperfResult: double getQuantile() */
union { null, double } quantile;
/** metrics-lib/TorperfResult: int getCircId() */
union { null, int } circ_id;
/** metrics-lib/TorperfResult: int getUsedBy() */
union { null, int } used_by;
}
}
> On Wed, Feb 24, 2016 at 7:46 AM, tl <tl...@rat.io> wrote:
> Hi again,
>
> I still haven’t found a solution to this problem. Does this look like some beginners Java mistake (because that may well be…)? Is it okay to ask the same question on stackoverflow or would that count as crossposting/spamming?
>
> Cheers,
> Thomas
>
>
> > On 23.02.2016, at 02:22, tl <tl...@rat.io> wrote:
> >
> > Hi,
> >
> >
> > I want to convert incoming data to Avro and JSON (and later Parquet). Avro conversion is working okay, but JSON conversion throws the following error that I don’t understand:
> >
> > Exception in thread "main" java.lang.RuntimeException: Not the Json schema: {"type":"record","name":"Torperf","namespace":"converTor.torperf","fields":[{"name":"descriptor_type","type":"string","default":"torperf 1.0"},
> > [ … omitted for brevity …]
> > {"name":"circ_id","type":["null","int"],"doc":"metrics-lib/TorperfResult: int getCircId()"},{"name":"used_by","type":["null","int"],"doc":"metrics-lib/TorperfResult: int getUsedBy()"}],"aliases":["torperfResult"]}
> > at org.apache.avro.data.Json$ObjectWriter.setSchema(Json.java:117)
> > at converTor.WriterObject.<init>(WriterObject.java:116)
> > at converTor.TypeWriter.get(TypeWriter.java:31)
> > at converTor.ConverTor.main(ConverTor.java:249)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> >
> > … but that schema is indeed the schema that I want to use.
> >
> >
> > This is a snippet of my code:
> >
> > File schemaFile = new File("schema/jsonSchema.avsc");
> > Schema.Parser parser = new Schema.Parser();
> > Schema mySchema = parser.parse(schemaFile) ;
> >
> > Json.ObjectWriter jsonDatumWriter = new Json.ObjectWriter();
> > jsonDatumWriter.setSchema(mySchema);
> > OutputStream out = new FileOutputStream(outputFile);
> > Encoder encoder = EncoderFactory.get().jsonEncoder(mySchema, out);
> >
> >
> > Can somebody give me a hint?
> >
> >
> > Thanks,
> > Thomas
>
>
>
>
>
>
>
> --
> --
> Cheers,
> Praj
< he not busy being born is busy dying >
Re: Json.ObjectWriter - "Not the Json schema"
Posted by tl <tl...@rat.io>.
> On 24.02.2016, at 19:08, Prajwal Tuladhar <pr...@infynyxx.com> wrote:
>
> Can you paste your avro IDL schema?
I had already crafted a more elaborate version of this question for stackoverflow and was just about to post it when you answered my request. Maybe it’s still helpful so I post it here. Thanks for taking the time!
I'm writing a tool to convert data from a homegrown format to Avro and JSON (and later Parquet), using Avro 1.8.0. Conversion to Avro is working okay, but JSON conversion throws the following error:
Exception in thread "main" java.lang.RuntimeException: Not the Json schema:
{"type":"record","name":"Torperf","namespace":"converTor.torperf",
"fields":[{"name":"descriptor_type","type":"string","
[... rest of the schema omitted for brevity]
Irritatingly this is the schema that I passed along and which indeed I want the converter to use. I have no idea what Avro is complaining about.
This is the relevant snippet of my code:
// parse the schema file
Schema.Parser parser = new Schema.Parser();
Schema mySchema;
// tried two ways to load the schema
// like this
File schemaFile = new File("myJsonSchema.avsc");
mySchema = parser.parse(schemaFile) ;
// and also like Json.class loads it's schema
mySchema = parser.parse(Json.class.getResourceAsStream("myJsonSchema.avsc"));
// initialize the writer
Json.ObjectWriter jsonDatumWriter = new Json.ObjectWriter();
jsonDatumWriter.setSchema(mySchema);
OutputStream out = new FileOutputStream(new File("output.avro"));
Encoder encoder = EncoderFactory.get().jsonEncoder(mySchema, out);
// append a record created by way of a specific mapping
jsonDatumWriter.write(specificRecord, encoder);
I tried some stuff like replacing my schema with the one returned from the exception, but to no avail (and apart from whitespace and linefeeds they are the same).
From staring at org.apache.avro.data.Json it seems to me like Avro is checking my record schema against it's own schema of a Json record (line 171) for equality (line 222).
36 public static final Schema SCHEMA;
171 SCHEMA = Schema.parse(Json.class.getResourceAsStream("/org/apache/avro/data/Json.avsc"));
221 public void setSchema(Schema schema) {
222 if(!Json.SCHEMA.equals(schema)) {
223 throw new RuntimeException("Not the Json schema: " + schema);
224 }
225 }
The referenced Json.avsc is short:
{"type": "record", "name": "Json", "namespace":"org.apache.avro.data",
"fields": [
{"name": "value",
"type": [
"long",
"double",
"string",
"boolean",
"null",
{"type": "array", "items": "Json"},
{"type": "map", "values": "Json"}
]
}
]
}
`equals` is implemented in org.apache.avro.Schema.class, line 256:
public boolean equals(Object o) {
if(o == this) {
return true;
} else if(!(o instanceof Schema)) {
return false;
} else {
Schema that = (Schema)o;
return this.type != that.type?false:this.equalCachedHash(that) && this.props.equals(that.props);
}
}
I don't fully understand what's going on in the third check (especially equalCachedHash()) but I only recognize checks for equality in a trivial sense and obviously my schema wouldn't model the JSON data model.
Also I can't find any examples or notes about usage of Avro's Json.ObjectWriter on the InterWebs. I wonder if I should go with the deprecated Json.Writer instead. Not that it's much better documented but there are at least a few code snippets online to learn and glean from.
The full source is available at https://github.com/tomlurge/converTor
> On Wed, Feb 24, 2016 at 7:46 AM, tl <tl...@rat.io> wrote:
> Hi again,
>
> I still haven’t found a solution to this problem. Does this look like some beginners Java mistake (because that may well be…)? Is it okay to ask the same question on stackoverflow or would that count as crossposting/spamming?
>
> Cheers,
> Thomas
>
>
> > On 23.02.2016, at 02:22, tl <tl...@rat.io> wrote:
> >
> > Hi,
> >
> >
> > I want to convert incoming data to Avro and JSON (and later Parquet). Avro conversion is working okay, but JSON conversion throws the following error that I don’t understand:
> >
> > Exception in thread "main" java.lang.RuntimeException: Not the Json schema: {"type":"record","name":"Torperf","namespace":"converTor.torperf","fields":[{"name":"descriptor_type","type":"string","default":"torperf 1.0"},
> > [ … omitted for brevity …]
> > {"name":"circ_id","type":["null","int"],"doc":"metrics-lib/TorperfResult: int getCircId()"},{"name":"used_by","type":["null","int"],"doc":"metrics-lib/TorperfResult: int getUsedBy()"}],"aliases":["torperfResult"]}
> > at org.apache.avro.data.Json$ObjectWriter.setSchema(Json.java:117)
> > at converTor.WriterObject.<init>(WriterObject.java:116)
> > at converTor.TypeWriter.get(TypeWriter.java:31)
> > at converTor.ConverTor.main(ConverTor.java:249)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> >
> > … but that schema is indeed the schema that I want to use.
> >
> >
> > This is a snippet of my code:
> >
> > File schemaFile = new File("schema/jsonSchema.avsc");
> > Schema.Parser parser = new Schema.Parser();
> > Schema mySchema = parser.parse(schemaFile) ;
> >
> > Json.ObjectWriter jsonDatumWriter = new Json.ObjectWriter();
> > jsonDatumWriter.setSchema(mySchema);
> > OutputStream out = new FileOutputStream(outputFile);
> > Encoder encoder = EncoderFactory.get().jsonEncoder(mySchema, out);
> >
> >
> > Can somebody give me a hint?
> >
> >
> > Thanks,
> > Thomas
>
>
>
>
>
>
>
> --
> --
> Cheers,
> Praj
< he not busy being born is busy dying >
Re: Json.ObjectWriter - "Not the Json schema"
Posted by Prajwal Tuladhar <pr...@infynyxx.com>.
Can you paste your avro IDL schema?
On Wed, Feb 24, 2016 at 7:46 AM, tl <tl...@rat.io> wrote:
> Hi again,
>
> I still haven’t found a solution to this problem. Does this look like some
> beginners Java mistake (because that may well be…)? Is it okay to ask the
> same question on stackoverflow or would that count as crossposting/spamming?
>
> Cheers,
> Thomas
>
>
> > On 23.02.2016, at 02:22, tl <tl...@rat.io> wrote:
> >
> > Hi,
> >
> >
> > I want to convert incoming data to Avro and JSON (and later Parquet).
> Avro conversion is working okay, but JSON conversion throws the following
> error that I don’t understand:
> >
> > Exception in thread "main" java.lang.RuntimeException: Not the Json
> schema:
> {"type":"record","name":"Torperf","namespace":"converTor.torperf","fields":[{"name":"descriptor_type","type":"string","default":"torperf
> 1.0"},
> > [ … omitted for brevity …]
> >
> {"name":"circ_id","type":["null","int"],"doc":"metrics-lib/TorperfResult:
> int
> getCircId()"},{"name":"used_by","type":["null","int"],"doc":"metrics-lib/TorperfResult:
> int getUsedBy()"}],"aliases":["torperfResult"]}
> > at org.apache.avro.data.Json$ObjectWriter.setSchema(Json.java:117)
> > at converTor.WriterObject.<init>(WriterObject.java:116)
> > at converTor.TypeWriter.get(TypeWriter.java:31)
> > at converTor.ConverTor.main(ConverTor.java:249)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> >
> > … but that schema is indeed the schema that I want to use.
> >
> >
> > This is a snippet of my code:
> >
> > File schemaFile = new File("schema/jsonSchema.avsc");
> > Schema.Parser parser = new Schema.Parser();
> > Schema mySchema = parser.parse(schemaFile) ;
> >
> > Json.ObjectWriter jsonDatumWriter = new Json.ObjectWriter();
> > jsonDatumWriter.setSchema(mySchema);
> > OutputStream out = new FileOutputStream(outputFile);
> > Encoder encoder = EncoderFactory.get().jsonEncoder(mySchema, out);
> >
> >
> > Can somebody give me a hint?
> >
> >
> > Thanks,
> > Thomas
>
>
>
>
>
--
--
Cheers,
Praj
Re: Json.ObjectWriter - "Not the Json schema"
Posted by tl <tl...@rat.io>.
Hi again,
I still haven’t found a solution to this problem. Does this look like some beginners Java mistake (because that may well be…)? Is it okay to ask the same question on stackoverflow or would that count as crossposting/spamming?
Cheers,
Thomas
> On 23.02.2016, at 02:22, tl <tl...@rat.io> wrote:
>
> Hi,
>
>
> I want to convert incoming data to Avro and JSON (and later Parquet). Avro conversion is working okay, but JSON conversion throws the following error that I don’t understand:
>
> Exception in thread "main" java.lang.RuntimeException: Not the Json schema: {"type":"record","name":"Torperf","namespace":"converTor.torperf","fields":[{"name":"descriptor_type","type":"string","default":"torperf 1.0"},
> [ … omitted for brevity …]
> {"name":"circ_id","type":["null","int"],"doc":"metrics-lib/TorperfResult: int getCircId()"},{"name":"used_by","type":["null","int"],"doc":"metrics-lib/TorperfResult: int getUsedBy()"}],"aliases":["torperfResult"]}
> at org.apache.avro.data.Json$ObjectWriter.setSchema(Json.java:117)
> at converTor.WriterObject.<init>(WriterObject.java:116)
> at converTor.TypeWriter.get(TypeWriter.java:31)
> at converTor.ConverTor.main(ConverTor.java:249)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
> … but that schema is indeed the schema that I want to use.
>
>
> This is a snippet of my code:
>
> File schemaFile = new File("schema/jsonSchema.avsc");
> Schema.Parser parser = new Schema.Parser();
> Schema mySchema = parser.parse(schemaFile) ;
>
> Json.ObjectWriter jsonDatumWriter = new Json.ObjectWriter();
> jsonDatumWriter.setSchema(mySchema);
> OutputStream out = new FileOutputStream(outputFile);
> Encoder encoder = EncoderFactory.get().jsonEncoder(mySchema, out);
>
>
> Can somebody give me a hint?
>
>
> Thanks,
> Thomas