You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Ayyappan Arasu <ay...@yahoo-inc.com> on 2012/11/27 19:11:09 UTC
Support for null in String primitive types
Hi,
I have an avro schema that I would like to use the default values as null for string types(primitives). Is there a way I can enable it or is this intentional? I am getting following error -
Error>
FAILED: testAvroToJson
org.apache.avro.AvroTypeException: Non-string default value for string: null
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:363)
at org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:178)
Avro Schema>
{
"schemas" : [
{
"namespace" : "com.mycompany",
"version" : "1.0",
"name" : "employee",
"type" : "record",
"fields" : [
{"name" : " _id", "type": "string"},
{"name" : "_name", "type" : "string" },
{"name" : "_email" , "type": "string", "default" : null }
]
}
]
}
Test Code>
private String avro2Json(Object avroData){
ByteArrayOutputStream output = new ByteArrayOutputStream();
GenericDatumWriter<Object> writer = new GenericDatumWriter<Object>(avroSchema);
String json = null;
try {
JsonEncoder jsonEncoder = EncoderFactory.get().jsonEncoder(avroSchema, output);
writer.write(avroData, jsonEncoder);
jsonEncoder.flush();
json = output.toString();
} catch (IOException ioe) {
logger.error("Error occured:" + ioe.getMessage());
} finally {
try {
output.close();
} catch (IOException e) {
// ignore
logger.error("Error occured:" + e.getMessage());
}
}
return json;
}
@Test
public void testAvroToJson(){
GenericRecordBuilder builder = new GenericRecordBuilder(avroSchema);
builder.set("_id", "1000")
.set("_name", "Dustin Hoffman")
String jsonStr = avro2Json(builder.build());
JsonNode node = null;
try{
node = mapper.readValue(jsonStr, JsonNode.class);
}catch(JsonParseException jpe){
Assert.fail("Exception occured when it should not have:" + jpe.getMessage());
}catch(JsonMappingException jme){
Assert.fail("Exception occured when it should not have:" + jme.getMessage());
}catch(IOException ioe){
Assert.fail("Exception occured when it should not have:" + ioe.getMessage());
}
Assert.assertNotNull(node);
Assert.assertEquals(node.get("_id").getValueAsText(), "1000");
}
It seems the failure is in public static void encode(Encoder e, Schema s, JsonNode n) in org.apache.avro.io.parsing.ResolvingGrammerGenerator
In the check below, the value of n is NullNode. Shouldn't String be nullable by default?
case STRING:
if (!n.isTextual())
throw new AvroTypeException("Non-string default value for string: "+n);
e.writeString(n.getTextValue());
break;
Please let me know if you have any pointers around this?
Thanks,
Ayyappan
Re: Support for null in String primitive types
Posted by Jeff Kolesky <je...@kolesky.com>.
I believe you need to set that type to be a union of types "null" and
"string" like so:
{"name" : "_email" , "type": ["null", "string"], "default" : null }
On Tue, Nov 27, 2012 at 10:11 AM, Ayyappan Arasu <ay...@yahoo-inc.com>wrote:
> Hi,
>
> I have an avro schema that I would like to use the default values as null
> for string types(primitives). Is there a way I can enable it or is this
> intentional? I am getting following error -
>
> Error>
>
> FAILED: testAvroToJson
> org.apache.avro.AvroTypeException: Non-string default value for string:
> null
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:363)
> at
> org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:178)
>
> Avro Schema>
>
> {
> "schemas" : [
> {
> "namespace" : "com.mycompany",
> "version" : "1.0",
> "name" : "employee",
> "type" : "record",
> "fields" : [
> {"name" : " _id", "type": "string"},
> {"name" : "_name", "type" : "string" },
> {"name" : "_email" , "type": "string", "default" :
> null }
> ]
> }
> ]
> }
>
> Test Code>
>
> private String avro2Json(Object avroData){
> ByteArrayOutputStream output = new ByteArrayOutputStream();
> GenericDatumWriter<Object> writer = new
> GenericDatumWriter<Object>(avroSchema);
> String json = null;
> try {
> JsonEncoder jsonEncoder =
> EncoderFactory.get().jsonEncoder(avroSchema, output);
> writer.write(avroData, jsonEncoder);
> jsonEncoder.flush();
> json = output.toString();
> } catch (IOException ioe) {
> logger.error("Error occured:" + ioe.getMessage());
> } finally {
> try {
> output.close();
> } catch (IOException e) {
> // ignore
> logger.error("Error occured:" + e.getMessage());
> }
> }
> return json;
> }
>
> @Test
> public void testAvroToJson(){
> GenericRecordBuilder builder = new
> GenericRecordBuilder(avroSchema);
> builder.set("_id", "1000")
> .set("_name", "Dustin Hoffman")
> String jsonStr = avro2Json(builder.build());
> JsonNode node = null;
> try{
> node = mapper.readValue(jsonStr, JsonNode.class);
> }catch(JsonParseException jpe){
> Assert.fail("Exception occured when it should not have:" +
> jpe.getMessage());
> }catch(JsonMappingException jme){
> Assert.fail("Exception occured when it should not have:" +
> jme.getMessage());
> }catch(IOException ioe){
> Assert.fail("Exception occured when it should not have:" +
> ioe.getMessage());
> }
> Assert.assertNotNull(node);
> Assert.assertEquals(node.get("_id").getValueAsText(), "1000");
> }
>
> It seems the failure is in public static void encode(Encoder e, Schema
> s, JsonNode n) in org.apache.avro.io.parsing.ResolvingGrammerGenerator
>
> In the check below, the value of n is NullNode. Shouldn't String be
> nullable by default?
>
> case STRING:
> if (!n.isTextual())
> throw new AvroTypeException("Non-string default value for string:
> "+n);
> e.writeString(n.getTextValue());
> break;
>
> Please let me know if you have any pointers around this?
>
> Thanks,
> Ayyappan
>
>
Re: Support for null in String primitive types
Posted by Jeff Kolesky <je...@gmail.com>.
I believe you need to set that type to be a union of types "null" and
"string" like so:
{"name" : "_email" , "type": ["null", "string"], "default" : null }
On Tue, Nov 27, 2012 at 10:11 AM, Ayyappan Arasu <ay...@yahoo-inc.com>wrote:
> Hi,
>
> I have an avro schema that I would like to use the default values as null
> for string types(primitives). Is there a way I can enable it or is this
> intentional? I am getting following error -
>
> Error>
>
> FAILED: testAvroToJson
> org.apache.avro.AvroTypeException: Non-string default value for string:
> null
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:363)
> at
> org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:178)
>
> Avro Schema>
>
> {
> "schemas" : [
> {
> "namespace" : "com.mycompany",
> "version" : "1.0",
> "name" : "employee",
> "type" : "record",
> "fields" : [
> {"name" : " _id", "type": "string"},
> {"name" : "_name", "type" : "string" },
> {"name" : "_email" , "type": "string", "default" :
> null }
> ]
> }
> ]
> }
>
> Test Code>
>
> private String avro2Json(Object avroData){
> ByteArrayOutputStream output = new ByteArrayOutputStream();
> GenericDatumWriter<Object> writer = new
> GenericDatumWriter<Object>(avroSchema);
> String json = null;
> try {
> JsonEncoder jsonEncoder =
> EncoderFactory.get().jsonEncoder(avroSchema, output);
> writer.write(avroData, jsonEncoder);
> jsonEncoder.flush();
> json = output.toString();
> } catch (IOException ioe) {
> logger.error("Error occured:" + ioe.getMessage());
> } finally {
> try {
> output.close();
> } catch (IOException e) {
> // ignore
> logger.error("Error occured:" + e.getMessage());
> }
> }
> return json;
> }
>
> @Test
> public void testAvroToJson(){
> GenericRecordBuilder builder = new
> GenericRecordBuilder(avroSchema);
> builder.set("_id", "1000")
> .set("_name", "Dustin Hoffman")
> String jsonStr = avro2Json(builder.build());
> JsonNode node = null;
> try{
> node = mapper.readValue(jsonStr, JsonNode.class);
> }catch(JsonParseException jpe){
> Assert.fail("Exception occured when it should not have:" +
> jpe.getMessage());
> }catch(JsonMappingException jme){
> Assert.fail("Exception occured when it should not have:" +
> jme.getMessage());
> }catch(IOException ioe){
> Assert.fail("Exception occured when it should not have:" +
> ioe.getMessage());
> }
> Assert.assertNotNull(node);
> Assert.assertEquals(node.get("_id").getValueAsText(), "1000");
> }
>
> It seems the failure is in public static void encode(Encoder e, Schema
> s, JsonNode n) in org.apache.avro.io.parsing.ResolvingGrammerGenerator
>
> In the check below, the value of n is NullNode. Shouldn't String be
> nullable by default?
>
> case STRING:
> if (!n.isTextual())
> throw new AvroTypeException("Non-string default value for string:
> "+n);
> e.writeString(n.getTextValue());
> break;
>
> Please let me know if you have any pointers around this?
>
> Thanks,
> Ayyappan
>
>