You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Pierre de Frém <th...@hotmail.com> on 2015/04/14 21:33:03 UTC
Schema default values in C++ implementation
Hello,
I'm using the avro c++ api (1.7.8) to serialize/deserialize avro files.
The schema I want to use is the following:
{
"name": "cpx",
"type": "record",
"fields": [
{"name": "numbername", "type": "string"},
{"name": "re", "type": "double"},
{"name": "im", "type": "double", "default": 0} ]
}I try to write programmatically the schema above in c++ in the following way:
#include <avro/Schema.hh>
static avro::Schema getMySchema() { avro::RecordSchema node0("cpx");
{ avro::Schema node1 = avro::StringSchema(); node0.addField("numbername", node1); } { avro::Schema node1 = avro::DoubleSchema(); node0.addField("re", node1); } { avro::Schema node1 = avro::DoubleSchema(); node0.addField("im", node1); }
return node0;}
Then I write the file using something like:
...
...
avro::DataFileWriter<ObjAvro> pDfw_m = new avro::DataFileWriter<ObjAvro>(filename_p, *(
new avro::ValidSchema(getMySchema());
...
...
const ObjAvro& avro_l = ...
pDfw_m.write(avro_l);
...
The problem is that the schema at the top of the output file created is different tothe input schema. It looks like:
{
"name": "cpx",
"type": "record",
"fields": [
{"name": "numbername", "type": "string"},
{"name": "re", "type": "double"},
{"name": "im", "type": "double"} ]
}
Is there a way to specify the default values directly in the c++ code above (in the method getMySchema)?
(So that the schema of my output file will match the one of my input file.)
After a look in the source code, I see where the defaultAttributes are stored for a record node, and what the method .addField does.
The method addField adds the name of the leaf to the Node member leafNameAttributes_ and the leaf to the Node member leafAttributes_, but I can't see a simple way to add default values to the NodeRecord member defaultValues (NodeImpl.hh).
Thanks in advance.
Regards.
Pierre
adding metadata (Python)
Posted by Marius <m....@googlemail.com>.
Hey,
i have a little problem:
I made a little program in Python that saves IDs and byte offsets in a
dictionary in order to be able to jump to a specific Datum (defined by
the ID) and read only that Datum (without the need to iterate over all
Datums). Now i want to add my dict somewhere to the avro file where i
can access it without having to search for it. My first thought was to
add it to the metadata, but i have to add the Datums first to know where
my byte offsets are and as far as i know you can only add metadata to a
file if there are no Datums in it.
Any suggestions how i can solve this problem?
Greetings Marius
Re: Schema default values in C++ implementation
Posted by Svante Karlsson <sv...@csi.se>.
I think you are hit by https://issues.apache.org/jira/browse/AVRO-1335
I recently extended the avrogen_cpp thing so it also generates the
following members to your class.
...
static inline const boost::uuids::uuid schema_hash() { static const
boost::uuids::uuid
_hash(boost::uuids::string_generator()("eea03bf9-7719-1af0-1dfb-e8049f677f7d"));
return _hash; }
static inline const char* schema_as_string() { return
"{\"type\":\"record\",\"name\":\"cpx\",\"fields\":[{\"name\":\"numbername\",\"type\":\"string\"},{\"name\":\"re\",\"type\":\"double\"},{\"name\":\"im\",\"type\":\"double\"}]}";
}
static const avro::ValidSchema valid_schema() { static const
avro::ValidSchema
_validSchema(avro::compileJsonSchemaFromString(schema_as_string())); return
_validSchema; }
...
As you can see the existing C++ code seems to loose the default values
somewhere and this of course also makes the schema hash unusable (in my
use-case)
/svante