You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by John Smith <le...@gmail.com> on 2016/01/06 22:11:20 UTC
custom attributes
Hello,
Im wondering if its allowed to define custom attributes as part of the AVRO
schema, e.g.
{
"type" :"record",
"name": "Test",
"fields": [
{
"name": "ROW_ID",
"title": "row identifier ",
"type": "integer",
* "description": "something",, "ValidityDate":
"2015-01-01", "ValidityOption": "Delete",*
},
...
Can I define those three new attributes *description, **ValidityDate,
**ValidityOption
*that extends description of the record "ROW_ID"?
Thank you
Re: custom attributes
Posted by John Smith <le...@gmail.com>.
Ok,
looks like my fail, the problem seems to be inside
https://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
When I try to dump data using Pig AvroStorage all the additional attributes
are removed from AVRO schema.
On Thu, Jan 7, 2016 at 10:53 AM, John Smith <le...@gmail.com> wrote:
> Hi,
>
> Im using parser() method (
> https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/Schema.Parser.html#parse%28java.lang.String%29).
>
>
> The question is if i have to write my own parser() method, extending the
> one from the AVRO api or if there is any other elegant solution to
> implement it.
>
>
> Thank you
>
>
> On Thu, Jan 7, 2016 at 8:11 AM, John Smith <le...@gmail.com> wrote:
>
>> Hi Peter,
>>
>> sorry it was mistype, it should be
>>
>> {
>> "type" :"record",
>> "name": "Test",
>> "fields": [
>> {
>> "name": "ROW_ID",
>> "title": "row identifier ",
>> "type": "integer",
>>
>>
>> * "description": "something",, "ValidityDate":
>> "2015-01-01", "ValidityOption": "Delete"*
>> },
>> ...
>>
>>
>> But how you add your own attributes, in your case "primarykey":"true”
>>
>>
>> Thanks
>>
>> On Thu, Jan 7, 2016 at 1:16 AM, Peter Lappo <pe...@gmail.com>
>> wrote:
>>
>>> Yes you can but be careful with the trailing comma - avro parser is
>>> quite strict and will fail
>>>
>>> *"ValidityOption": "Delete",*
>>>
>>>
>>> I use
>>> {"name":"CUST_ID","type":"string","primarykey":"true”}
>>> and define the custom attribute in the avsc file
>>>
>>>
>>> On 6 Jan 2016, at 21:11, John Smith <le...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> Im wondering if its allowed to define custom attributes as part of the
>>> AVRO schema, e.g.
>>>
>>>
>>> {
>>> "type" :"record",
>>> "name": "Test",
>>> "fields": [
>>> {
>>> "name": "ROW_ID",
>>> "title": "row identifier ",
>>> "type": "integer",
>>>
>>>
>>> * "description": "something",, "ValidityDate":
>>> "2015-01-01", "ValidityOption": "Delete",*
>>> },
>>> ...
>>>
>>>
>>> Can I define those three new attributes *description, **ValidityDate, **ValidityOption
>>> *that extends description of the record "ROW_ID"?
>>>
>>>
>>> Thank you
>>>
>>>
>>>
>>>
>>
>
Re: custom attributes
Posted by John Smith <le...@gmail.com>.
Hi,
Im using parser() method (
https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/Schema.Parser.html#parse%28java.lang.String%29).
The question is if i have to write my own parser() method, extending the
one from the AVRO api or if there is any other elegant solution to
implement it.
Thank you
On Thu, Jan 7, 2016 at 8:11 AM, John Smith <le...@gmail.com> wrote:
> Hi Peter,
>
> sorry it was mistype, it should be
>
> {
> "type" :"record",
> "name": "Test",
> "fields": [
> {
> "name": "ROW_ID",
> "title": "row identifier ",
> "type": "integer",
>
>
> * "description": "something",, "ValidityDate":
> "2015-01-01", "ValidityOption": "Delete"*
> },
> ...
>
>
> But how you add your own attributes, in your case "primarykey":"true”
>
>
> Thanks
>
> On Thu, Jan 7, 2016 at 1:16 AM, Peter Lappo <pe...@gmail.com> wrote:
>
>> Yes you can but be careful with the trailing comma - avro parser is quite
>> strict and will fail
>>
>> *"ValidityOption": "Delete",*
>>
>>
>> I use
>> {"name":"CUST_ID","type":"string","primarykey":"true”}
>> and define the custom attribute in the avsc file
>>
>>
>> On 6 Jan 2016, at 21:11, John Smith <le...@gmail.com> wrote:
>>
>> Hello,
>>
>> Im wondering if its allowed to define custom attributes as part of the
>> AVRO schema, e.g.
>>
>>
>> {
>> "type" :"record",
>> "name": "Test",
>> "fields": [
>> {
>> "name": "ROW_ID",
>> "title": "row identifier ",
>> "type": "integer",
>>
>>
>> * "description": "something",, "ValidityDate":
>> "2015-01-01", "ValidityOption": "Delete",*
>> },
>> ...
>>
>>
>> Can I define those three new attributes *description, **ValidityDate, **ValidityOption
>> *that extends description of the record "ROW_ID"?
>>
>>
>> Thank you
>>
>>
>>
>>
>
Re: custom attributes
Posted by John Smith <le...@gmail.com>.
Hi Peter,
sorry it was mistype, it should be
{
"type" :"record",
"name": "Test",
"fields": [
{
"name": "ROW_ID",
"title": "row identifier ",
"type": "integer",
* "description": "something",, "ValidityDate":
"2015-01-01", "ValidityOption": "Delete"*
},
...
But how you add your own attributes, in your case "primarykey":"true”
Thanks
On Thu, Jan 7, 2016 at 1:16 AM, Peter Lappo <pe...@gmail.com> wrote:
> Yes you can but be careful with the trailing comma - avro parser is quite
> strict and will fail
>
> *"ValidityOption": "Delete",*
>
>
> I use
> {"name":"CUST_ID","type":"string","primarykey":"true”}
> and define the custom attribute in the avsc file
>
>
> On 6 Jan 2016, at 21:11, John Smith <le...@gmail.com> wrote:
>
> Hello,
>
> Im wondering if its allowed to define custom attributes as part of the
> AVRO schema, e.g.
>
>
> {
> "type" :"record",
> "name": "Test",
> "fields": [
> {
> "name": "ROW_ID",
> "title": "row identifier ",
> "type": "integer",
>
>
> * "description": "something",, "ValidityDate":
> "2015-01-01", "ValidityOption": "Delete",*
> },
> ...
>
>
> Can I define those three new attributes *description, **ValidityDate, **ValidityOption
> *that extends description of the record "ROW_ID"?
>
>
> Thank you
>
>
>
>
Re: custom attributes
Posted by Peter Lappo <pe...@gmail.com>.
Yes you can but be careful with the trailing comma - avro parser is quite strict and will fail
> "ValidityOption": "Delete",
I use
{"name":"CUST_ID","type":"string","primarykey":"true”}
and define the custom attribute in the avsc file
> On 6 Jan 2016, at 21:11, John Smith <le...@gmail.com> wrote:
>
> Hello,
>
> Im wondering if its allowed to define custom attributes as part of the AVRO schema, e.g.
>
>
> {
> "type" :"record",
> "name": "Test",
> "fields": [
> {
> "name": "ROW_ID",
> "title": "row identifier ",
> "type": "integer",
> "description": "something",,
> "ValidityDate": "2015-01-01",
> "ValidityOption": "Delete",
> },
> ...
>
>
> Can I define those three new attributes description, ValidityDate, ValidityOption that extends description of the record "ROW_ID"?
>
>
> Thank you
>
>