You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Matt Pouttu-Clarke <Ma...@icrossing.com> on 2011/06/10 19:46:07 UTC
New name requirements for Avro 1.5
Hi All,
I am using Avro for processing/encoding billions of rows using Hadoop. When
I attempted to upgrade to Avro 1.5 I got an error about illegal field names.
I notice this case happens when I use a O-¹ character or a O.¹ in a field
name.
Are there any other restrictions on fields names I need to know when
upgrading?
Thanks,
Matt
iCrossing Privileged and Confidential Information
This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: New name requirements for Avro 1.5
Posted by Doug Cutting <cu...@apache.org>.
On 06/10/2011 09:32 PM, Matt Pouttu-Clarke wrote:
> Can we get away with "tweeking" the field name in the JSON meta data portion
> of the file, or do we have to re-write the whole file again with the new
> meta data?
Yes, you should be able to change that one character in the binary file
and then read the file. But a better compatibility fix would be good.
Doug
Re: New name requirements for Avro 1.5
Posted by Ken Krugler <kk...@transpac.com>.
On Jun 10, 2011, at 4:01pm, Scott Carey wrote:
> Ok, there was a bug in 1.4 that allowed for invalid names. Sounds like we
> need something that allows reading some older files.
> Please open a JIRA ticket explaining the issue, we can discuss
> solutions/workarounds there.
https://issues.apache.org/jira/browse/AVRO-838
Thanks,
-- Ken
>
> On 6/10/11 1:48 PM, "Ken Krugler" <kk...@transpac.com> wrote:
>
>> I'd run into a similar issue, where I used a record name with a "-" in it
>> when creating files with Avro 1.4.
>>
>> So I've got files that are valid/readable with 1.4, but seem to be
>> unusable with 1.5.
>>
>> -- Ken
>>
>> On Jun 10, 2011, at 12:32pm, Matt Pouttu-Clarke wrote:
>>
>>> OK, thanks. This is a change from Avro 1.4 which will make older Avro
>>> files
>>> unreadable by 1.5 until the field names are changed.
>>>
>>> Can we get away with "tweeking" the field name in the JSON meta data
>>> portion
>>> of the file, or do we have to re-write the whole file again with the new
>>> meta data?
>>>
>>> On 6/10/11 12:23 PM, "Douglas Creager" <dc...@dcreager.net> wrote:
>>>
>>>>> I am using Avro for processing/encoding billions of rows using
>>>>> Hadoop. When
>>>>> I attempted to upgrade to Avro 1.5 I got an error about illegal field
>>>>> names.
>>>>> I notice this case happens when I use a O-¹ character or a O.¹ in a
>>>>> field
>>>>> name.
>>>>>
>>>>> Are there any other restrictions on fields names I need to know when
>>>>> upgrading?
>>>>
>>>> You can find the full details in the Avro spec [1]. It's basically
>>>> just like
>>>> the identifier restrictions in most programming languages. Names can
>>>> only
>>>> contain letters, numbers, and underscores, and can't start with a
>>>> number.
>>>> Dots are used to created namespaces, just like fully-qualified Java
>>>> identifiers.
>>>>
>>>> [1] http://avro.apache.org/docs/current/spec.html
>>>>
>>>> cheers
>>>> doug
>>>
>>>
>>> iCrossing Privileged and Confidential Information
>>> This email message is for the sole use of the intended recipient(s) and
>>> may contain confidential and privileged information of iCrossing. Any
>>> unauthorized review, use, disclosure or distribution is prohibited. If
>>> you are not the intended recipient, please contact the sender by reply
>>> email and destroy all copies of the original message.
>>>
>>>
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://bixolabs.com
>> custom data mining solutions
>>
>>
>>
>>
>>
>>
>
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions
Re: New name requirements for Avro 1.5
Posted by Scott Carey <sc...@richrelevance.com>.
Ok, there was a bug in 1.4 that allowed for invalid names. Sounds like we
need something that allows reading some older files.
Please open a JIRA ticket explaining the issue, we can discuss
solutions/workarounds there.
On 6/10/11 1:48 PM, "Ken Krugler" <kk...@transpac.com> wrote:
>I'd run into a similar issue, where I used a record name with a "-" in it
>when creating files with Avro 1.4.
>
>So I've got files that are valid/readable with 1.4, but seem to be
>unusable with 1.5.
>
>-- Ken
>
>On Jun 10, 2011, at 12:32pm, Matt Pouttu-Clarke wrote:
>
>> OK, thanks. This is a change from Avro 1.4 which will make older Avro
>>files
>> unreadable by 1.5 until the field names are changed.
>>
>> Can we get away with "tweeking" the field name in the JSON meta data
>>portion
>> of the file, or do we have to re-write the whole file again with the new
>> meta data?
>>
>> On 6/10/11 12:23 PM, "Douglas Creager" <dc...@dcreager.net> wrote:
>>
>>>> I am using Avro for processing/encoding billions of rows using
>>>>Hadoop. When
>>>> I attempted to upgrade to Avro 1.5 I got an error about illegal field
>>>>names.
>>>> I notice this case happens when I use a O-¹ character or a O.¹ in a
>>>>field
>>>> name.
>>>>
>>>> Are there any other restrictions on fields names I need to know when
>>>> upgrading?
>>>
>>> You can find the full details in the Avro spec [1]. It's basically
>>>just like
>>> the identifier restrictions in most programming languages. Names can
>>>only
>>> contain letters, numbers, and underscores, and can't start with a
>>>number.
>>> Dots are used to created namespaces, just like fully-qualified Java
>>> identifiers.
>>>
>>> [1] http://avro.apache.org/docs/current/spec.html
>>>
>>> cheers
>>> doug
>>
>>
>> iCrossing Privileged and Confidential Information
>> This email message is for the sole use of the intended recipient(s) and
>>may contain confidential and privileged information of iCrossing. Any
>>unauthorized review, use, disclosure or distribution is prohibited. If
>>you are not the intended recipient, please contact the sender by reply
>>email and destroy all copies of the original message.
>>
>>
>
>--------------------------
>Ken Krugler
>+1 530-210-6378
>http://bixolabs.com
>custom data mining solutions
>
>
>
>
>
>
Re: New name requirements for Avro 1.5
Posted by Ken Krugler <kk...@transpac.com>.
I'd run into a similar issue, where I used a record name with a "-" in it when creating files with Avro 1.4.
So I've got files that are valid/readable with 1.4, but seem to be unusable with 1.5.
-- Ken
On Jun 10, 2011, at 12:32pm, Matt Pouttu-Clarke wrote:
> OK, thanks. This is a change from Avro 1.4 which will make older Avro files
> unreadable by 1.5 until the field names are changed.
>
> Can we get away with "tweeking" the field name in the JSON meta data portion
> of the file, or do we have to re-write the whole file again with the new
> meta data?
>
> On 6/10/11 12:23 PM, "Douglas Creager" <dc...@dcreager.net> wrote:
>
>>> I am using Avro for processing/encoding billions of rows using Hadoop. When
>>> I attempted to upgrade to Avro 1.5 I got an error about illegal field names.
>>> I notice this case happens when I use a O-’ character or a O.’ in a field
>>> name.
>>>
>>> Are there any other restrictions on fields names I need to know when
>>> upgrading?
>>
>> You can find the full details in the Avro spec [1]. It's basically just like
>> the identifier restrictions in most programming languages. Names can only
>> contain letters, numbers, and underscores, and can't start with a number.
>> Dots are used to created namespaces, just like fully-qualified Java
>> identifiers.
>>
>> [1] http://avro.apache.org/docs/current/spec.html
>>
>> cheers
>> –doug
>
>
> iCrossing Privileged and Confidential Information
> This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
>
>
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions
Re: New name requirements for Avro 1.5
Posted by Matt Pouttu-Clarke <Ma...@icrossing.com>.
OK, thanks. This is a change from Avro 1.4 which will make older Avro files
unreadable by 1.5 until the field names are changed.
Can we get away with "tweeking" the field name in the JSON meta data portion
of the file, or do we have to re-write the whole file again with the new
meta data?
On 6/10/11 12:23 PM, "Douglas Creager" <dc...@dcreager.net> wrote:
>> I am using Avro for processing/encoding billions of rows using Hadoop. When
>> I attempted to upgrade to Avro 1.5 I got an error about illegal field names.
>> I notice this case happens when I use a O-¹ character or a O.¹ in a field
>> name.
>>
>> Are there any other restrictions on fields names I need to know when
>> upgrading?
>
> You can find the full details in the Avro spec [1]. It's basically just like
> the identifier restrictions in most programming languages. Names can only
> contain letters, numbers, and underscores, and can't start with a number.
> Dots are used to created namespaces, just like fully-qualified Java
> identifiers.
>
> [1] http://avro.apache.org/docs/current/spec.html
>
> cheers
> doug
iCrossing Privileged and Confidential Information
This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: New name requirements for Avro 1.5
Posted by Douglas Creager <dc...@dcreager.net>.
> I am using Avro for processing/encoding billions of rows using Hadoop. When I attempted to upgrade to Avro 1.5 I got an error about illegal field names. I notice this case happens when I use a ‘-’ character or a ‘.’ in a field name.
>
> Are there any other restrictions on fields names I need to know when upgrading?
You can find the full details in the Avro spec [1]. It's basically just like the identifier restrictions in most programming languages. Names can only contain letters, numbers, and underscores, and can't start with a number. Dots are used to created namespaces, just like fully-qualified Java identifiers.
[1] http://avro.apache.org/docs/current/spec.html
cheers
–doug