You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Pierre de Frém <th...@hotmail.com> on 2015/04/14 21:33:03 UTC

Schema default values in C++ implementation

Hello,
I'm using the avro c++ api (1.7.8) to serialize/deserialize avro files.
The schema I want to use is the following:
{
    "name": "cpx",
    "type": "record",
    "fields": [
        {"name": "numbername", "type": "string"},
        {"name": "re", "type": "double"},
        {"name": "im", "type": "double", "default": 0}     ]
}I try to write programmatically the schema above in c++ in the following way:
#include <avro/Schema.hh>

static avro::Schema getMySchema() {    avro::RecordSchema node0("cpx");
    {        avro::Schema node1 = avro::StringSchema();        node0.addField("numbername", node1);    }    {        avro::Schema node1 = avro::DoubleSchema();        node0.addField("re", node1);    }    {        avro::Schema node1 = avro::DoubleSchema();        node0.addField("im", node1);    }
    return node0;}
Then I write the file using something like:
...
...
avro::DataFileWriter<ObjAvro> pDfw_m = new avro::DataFileWriter<ObjAvro>(filename_p, *(
                       new avro::ValidSchema(getMySchema());
...
...
const ObjAvro& avro_l = ...
pDfw_m.write(avro_l);
...


The problem is that the schema at the top of the output file created is different tothe input schema. It looks like:
{
    "name": "cpx",
    "type": "record",
    "fields": [
        {"name": "numbername", "type": "string"},
        {"name": "re", "type": "double"},
        {"name": "im", "type": "double"}     ]
}
Is there a way to specify the default values directly in the c++ code above (in the method getMySchema)? 
(So that the schema of my output file will match the one of my input file.)
After a look in the source code, I see where the defaultAttributes are stored for a record node, and what the method .addField does.
The method addField adds the name of the leaf to the Node member leafNameAttributes_ and the leaf to the Node member leafAttributes_, but I can't see a simple way to add default values to the NodeRecord member defaultValues (NodeImpl.hh).
Thanks in advance.
Regards.
Pierre
 		 	   		   		 	   		  

adding metadata (Python)

Posted by Marius <m....@googlemail.com>.
Hey,

i have a little problem:
I made a little program in Python that saves IDs and byte offsets in a 
dictionary in order to be able to jump to a specific Datum (defined by 
the ID) and read only that Datum (without the need to iterate over all 
Datums). Now i want to add my dict somewhere to the avro file where i 
can access it without having to search for it. My first thought was to 
add it to the metadata, but i have to add the Datums first to know where 
my byte offsets are and as far as i know you can only add metadata to a 
file if there are no Datums in it.

Any suggestions how i can solve this problem?

Greetings Marius

Re: Schema default values in C++ implementation

Posted by Svante Karlsson <sv...@csi.se>.
I think you are hit by https://issues.apache.org/jira/browse/AVRO-1335

I recently extended the avrogen_cpp thing so it also generates the
following members to your class.

...
   static inline const boost::uuids::uuid schema_hash()      { static const
boost::uuids::uuid
_hash(boost::uuids::string_generator()("eea03bf9-7719-1af0-1dfb-e8049f677f7d"));
return _hash; }
   static inline const char*              schema_as_string() { return
"{\"type\":\"record\",\"name\":\"cpx\",\"fields\":[{\"name\":\"numbername\",\"type\":\"string\"},{\"name\":\"re\",\"type\":\"double\"},{\"name\":\"im\",\"type\":\"double\"}]}";
}
   static const avro::ValidSchema         valid_schema()     { static const
avro::ValidSchema
_validSchema(avro::compileJsonSchemaFromString(schema_as_string())); return
_validSchema; }
...

As you can see the existing C++ code seems to loose the default values
somewhere and this of course also makes the schema hash unusable (in my
use-case)

/svante