You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Tianxiang Xiong <ti...@fundingcircle.com> on 2016/11/22 17:58:55 UTC
UUIDs in Avro Schemas
The AVRO-1554 <https://issues.apache.org/jira/browse/AVRO-1554> ticket
seems to suggest that the issue of representing UUIDs is resolved with
logical
types <http://avro.apache.org/docs/1.8.1/spec.html#Logical+Types> in Avro
1.8.1.
The specification offers several examples of using logical types; decimals
are represented as:
{
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 2
}
However, there does not seem to be an example of UUID. The most
popular answers <http://stackoverflow.com/a/19705149/864684> via
Google/StackOverflow are a few years old and do not make use of
logical types, and can therefore be misleading for newer versions of
Avro
Is there an idiomatic way of representing UUIDs in Avro Schemas?
Thanks,
--
*Tianxiang Xiong *| Clojure Engineer
*tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
747 Front Street, Floor 4 | San Francisco, CA 94111
Re: UUIDs in Avro Schemas
Posted by Tianxiang Xiong <ti...@fundingcircle.com>.
Are the listed logical types those that the Avro parser supports "out of
the box", i.e. if using the Avro 1.8.1 Java API, a logical date will be
automatically deserialized into a *java.util.Date* and vice-versa?
I see that the *LogicalTypes* class has a *uuid()* method, but there is no
corresponding *LogicalTypes.UUID* class. There's also no mention of a
logical UUID in the documentation
<https://avro.apache.org/docs/1.8.1/spec.html#Logical+Types>. What, then,
does the method do?
Is there a more detailed example of specifying a custom logical type? I
assume that after specifying a logical type in the JSON schema, we can
extend the parser so it can do the conversion automatically. There's an
example <https://github.com/mtth/avsc/wiki/Advanced-usage#logical-types> of
this for *avsc*, an Avro JavaScript library, but I haven't seen one for
Java.
On 23 November 2016 at 07:37, Sam Groth <sg...@yahoo-inc.com> wrote:
> For UUIDs, I would probably use the fixed type, but a string would also
> work. See the specification for the different types that avro supports:
> http://avro.apache.org/docs/current/spec.html
>
>
> Sam
>
>
> On Tuesday, November 22, 2016 6:32 PM, Tianxiang Xiong <tianxiang.xiong@
> fundingcircle.com> wrote:
>
>
> I see, I'll close the ticket then if using logical types this way to
> specify UUID is idiomatic.
>
> On 22 November 2016 at 16:21, Yibing Shi <ys...@cloudera.com> wrote:
>
> Hi Tianxiang,
>
> I am sorry! I replied too quickly!
>
> I thought that you were complaining that you couldn't generate the UUID
> type with IDL compiler, but when I check the JIRA you have just created, I
> find that you actually want to know how to specify UUID type in JSON
> format, which has already been there. Please use below definition:
>
> {
> "type": "string",
> "logicalType": "uuid"
> }
>
> *Yibing Shi*
> *Customer Operations Engineer*
> <http://www.cloudera.com/>
>
> On Wed, Nov 23, 2016 at 10:43 AM, Tianxiang Xiong <tianxiang.xiong@
> fundingcircle.com <ti...@fundingcircle.com>> wrote:
>
> Thanks Yibing, I have created AVRO-1962
> <https://issues.apache.org/jira/browse/AVRO-1962>. Take a look and let me
> know if additional information/changes are needed to the ticket.
>
> On 22 November 2016 at 15:12, Yibing Shi <ys...@cloudera.com> wrote:
>
> Hi Tianxiang,
>
> Looks like that the UUID type hasn't been added to IDL definition yet.
>
> https://github.com/apache/avro /blob/master/lang/java/compile
> r/src/main/javacc/org/apache/a vro/compiler/idl/idl.jj#L214-L 244
> <https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/javacc/org/apache/avro/compiler/idl/idl.jj#L214-L244>
>
> Please create a JIRA to track this.
>
> *Yibing Shi*
> *Customer Operations Engineer*
> <http://www.cloudera.com/>
>
> On Wed, Nov 23, 2016 at 4:58 AM, Tianxiang Xiong <tianxiang.xiong@fundingcircle
> .com <ti...@fundingcircle.com>> wrote:
>
> The AVRO-1554 <https://issues.apache.org/jir a/browse/AVRO-1554
> <https://issues.apache.org/jira/browse/AVRO-1554>> ticket
> seems to suggest that the issue of representing UUIDs is resolved with
> logical
> types <http://avro.apache.org/docs/1 .8.1/spec.html#Logical+Types
> <http://avro.apache.org/docs/1.8.1/spec.html#Logical+Types>> in Avro
> 1.8.1.
>
> The specification offers several examples of using logical types; decimals
> are represented as:
>
> {
>
> "type": "bytes",
> "logicalType": "decimal",
> "precision": 4,
> "scale": 2
> }
>
> However, there does not seem to be an example of UUID. The most
> popular answers <http://stackoverflow.com/a/19 705149/864684
> <http://stackoverflow.com/a/19705149/864684>> via
> Google/StackOverflow are a few years old and do not make use of
> logical types, and can therefore be misleading for newer versions of
> Avro
>
> Is there an idiomatic way of representing UUIDs in Avro Schemas?
>
> Thanks,
>
> --
> *Tianxiang Xiong *| Clojure Engineer
> *tianxiang.xiong@fundingcircle. com <ti...@fundingcircle.com>*
> 747 Front Street, Floor 4 | San Francisco, CA 94111
>
>
>
>
>
> --
> *Tianxiang Xiong *| Clojure Engineer
> *tianxiang.xiong@fundingcircle. com <ti...@fundingcircle.com>*
> 747 Front Street, Floor 4 | San Francisco, CA 94111
>
>
>
>
>
> --
> *Tianxiang Xiong *| Clojure Engineer
> *tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
> 747 Front Street, Floor 4 | San Francisco, CA 94111
>
>
>
--
*Tianxiang Xiong *| Clojure Engineer
*tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
747 Front Street, Floor 4 | San Francisco, CA 94111
Re: UUIDs in Avro Schemas
Posted by Sam Groth <sg...@yahoo-inc.com>.
For UUIDs, I would probably use the fixed type, but a string would also work. See the specification for the different types that avro supports: http://avro.apache.org/docs/current/spec.html
Sam
On Tuesday, November 22, 2016 6:32 PM, Tianxiang Xiong <ti...@fundingcircle.com> wrote:
I see, I'll close the ticket then if using logical types this way to specify UUID is idiomatic.
On 22 November 2016 at 16:21, Yibing Shi <ys...@cloudera.com> wrote:
Hi Tianxiang,
I am sorry! I replied too quickly!
I thought that you were complaining that you couldn't generate the UUID type with IDL compiler, but when I check the JIRA you have just created, I find that you actually want to know how to specify UUID type in JSON format, which has already been there. Please use below definition:
{
"type": "string",
"logicalType": "uuid"
}
Yibing Shi
Customer Operations Engineer
On Wed, Nov 23, 2016 at 10:43 AM, Tianxiang Xiong <tianxiang.xiong@ fundingcircle.com> wrote:
Thanks Yibing, I have created AVRO-1962. Take a look and let me know if additional information/changes are needed to the ticket.
On 22 November 2016 at 15:12, Yibing Shi <ys...@cloudera.com> wrote:
Hi Tianxiang,
Looks like that the UUID type hasn't been added to IDL definition yet.
https://github.com/apache/avro /blob/master/lang/java/compile r/src/main/javacc/org/apache/a vro/compiler/idl/idl.jj#L214-L 244
Please create a JIRA to track this.
Yibing Shi
Customer Operations Engineer
On Wed, Nov 23, 2016 at 4:58 AM, Tianxiang Xiong <tianxiang.xiong@fundingcircle .com> wrote:
The AVRO-1554 <https://issues.apache.org/jir a/browse/AVRO-1554> ticket
seems to suggest that the issue of representing UUIDs is resolved with logical
types <http://avro.apache.org/docs/1 .8.1/spec.html#Logical+Types> in Avro
1.8.1.
The specification offers several examples of using logical types; decimals
are represented as:
{
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 2
}
However, there does not seem to be an example of UUID. The most
popular answers <http://stackoverflow.com/a/19 705149/864684> via
Google/StackOverflow are a few years old and do not make use of
logical types, and can therefore be misleading for newer versions of
Avro
Is there an idiomatic way of representing UUIDs in Avro Schemas?
Thanks,
--
Tianxiang Xiong | Clojure Engineer tianxiang.xiong@fundingcircle. com747 Front Street, Floor 4 | San Francisco, CA 94111
--
Tianxiang Xiong | Clojure Engineer tianxiang.xiong@fundingcircle. com747 Front Street, Floor 4 | San Francisco, CA 94111
--
Tianxiang Xiong | Clojure Engineer tianxiang.xiong@fundingcircle.com747 Front Street, Floor 4 | San Francisco, CA 94111
Re: UUIDs in Avro Schemas
Posted by Tianxiang Xiong <ti...@fundingcircle.com>.
I see, I'll close the ticket then if using logical types this way to
specify UUID is idiomatic.
On 22 November 2016 at 16:21, Yibing Shi <ys...@cloudera.com> wrote:
> Hi Tianxiang,
>
> I am sorry! I replied too quickly!
>
> I thought that you were complaining that you couldn't generate the UUID
> type with IDL compiler, but when I check the JIRA you have just created, I
> find that you actually want to know how to specify UUID type in JSON
> format, which has already been there. Please use below definition:
>
> {
> "type": "string",
> "logicalType": "uuid"
> }
>
> *Yibing Shi*
> *Customer Operations Engineer*
> <http://www.cloudera.com>
>
> On Wed, Nov 23, 2016 at 10:43 AM, Tianxiang Xiong <tianxiang.xiong@
> fundingcircle.com> wrote:
>
>> Thanks Yibing, I have created AVRO-1962
>> <https://issues.apache.org/jira/browse/AVRO-1962>. Take a look and let
>> me know if additional information/changes are needed to the ticket.
>>
>> On 22 November 2016 at 15:12, Yibing Shi <ys...@cloudera.com> wrote:
>>
>>> Hi Tianxiang,
>>>
>>> Looks like that the UUID type hasn't been added to IDL definition yet.
>>>
>>> https://github.com/apache/avro/blob/master/lang/java/compile
>>> r/src/main/javacc/org/apache/avro/compiler/idl/idl.jj#L214-L244
>>>
>>> Please create a JIRA to track this.
>>>
>>> *Yibing Shi*
>>> *Customer Operations Engineer*
>>> <http://www.cloudera.com>
>>>
>>> On Wed, Nov 23, 2016 at 4:58 AM, Tianxiang Xiong <
>>> tianxiang.xiong@fundingcircle.com> wrote:
>>>
>>>> The AVRO-1554 <https://issues.apache.org/jira/browse/AVRO-1554> ticket
>>>> seems to suggest that the issue of representing UUIDs is resolved with
>>>> logical
>>>> types <http://avro.apache.org/docs/1.8.1/spec.html#Logical+Types> in
>>>> Avro
>>>> 1.8.1.
>>>>
>>>> The specification offers several examples of using logical types;
>>>> decimals
>>>> are represented as:
>>>>
>>>> {
>>>>
>>>> "type": "bytes",
>>>> "logicalType": "decimal",
>>>> "precision": 4,
>>>> "scale": 2
>>>> }
>>>>
>>>> However, there does not seem to be an example of UUID. The most
>>>> popular answers <http://stackoverflow.com/a/19705149/864684> via
>>>> Google/StackOverflow are a few years old and do not make use of
>>>> logical types, and can therefore be misleading for newer versions of
>>>> Avro
>>>>
>>>> Is there an idiomatic way of representing UUIDs in Avro Schemas?
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>>
>>>> *Tianxiang Xiong *| Clojure Engineer
>>>>
>>>> *tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
>>>>
>>>> 747 Front Street, Floor 4 | San Francisco, CA 94111
>>>>
>>>
>>>
>>
>>
>> --
>>
>> *Tianxiang Xiong *| Clojure Engineer
>>
>> *tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
>>
>> 747 Front Street, Floor 4 | San Francisco, CA 94111
>>
>
>
--
*Tianxiang Xiong *| Clojure Engineer
*tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
747 Front Street, Floor 4 | San Francisco, CA 94111
Re: UUIDs in Avro Schemas
Posted by Yibing Shi <ys...@cloudera.com>.
Hi Tianxiang,
I am sorry! I replied too quickly!
I thought that you were complaining that you couldn't generate the UUID
type with IDL compiler, but when I check the JIRA you have just created, I
find that you actually want to know how to specify UUID type in JSON
format, which has already been there. Please use below definition:
{
"type": "string",
"logicalType": "uuid"
}
*Yibing Shi*
*Customer Operations Engineer*
<http://www.cloudera.com>
On Wed, Nov 23, 2016 at 10:43 AM, Tianxiang Xiong <
tianxiang.xiong@fundingcircle.com> wrote:
> Thanks Yibing, I have created AVRO-1962
> <https://issues.apache.org/jira/browse/AVRO-1962>. Take a look and let me
> know if additional information/changes are needed to the ticket.
>
> On 22 November 2016 at 15:12, Yibing Shi <ys...@cloudera.com> wrote:
>
>> Hi Tianxiang,
>>
>> Looks like that the UUID type hasn't been added to IDL definition yet.
>>
>> https://github.com/apache/avro/blob/master/lang/java/compile
>> r/src/main/javacc/org/apache/avro/compiler/idl/idl.jj#L214-L244
>>
>> Please create a JIRA to track this.
>>
>> *Yibing Shi*
>> *Customer Operations Engineer*
>> <http://www.cloudera.com>
>>
>> On Wed, Nov 23, 2016 at 4:58 AM, Tianxiang Xiong <
>> tianxiang.xiong@fundingcircle.com> wrote:
>>
>>> The AVRO-1554 <https://issues.apache.org/jira/browse/AVRO-1554> ticket
>>> seems to suggest that the issue of representing UUIDs is resolved with
>>> logical
>>> types <http://avro.apache.org/docs/1.8.1/spec.html#Logical+Types> in
>>> Avro
>>> 1.8.1.
>>>
>>> The specification offers several examples of using logical types;
>>> decimals
>>> are represented as:
>>>
>>> {
>>>
>>> "type": "bytes",
>>> "logicalType": "decimal",
>>> "precision": 4,
>>> "scale": 2
>>> }
>>>
>>> However, there does not seem to be an example of UUID. The most
>>> popular answers <http://stackoverflow.com/a/19705149/864684> via
>>> Google/StackOverflow are a few years old and do not make use of
>>> logical types, and can therefore be misleading for newer versions of
>>> Avro
>>>
>>> Is there an idiomatic way of representing UUIDs in Avro Schemas?
>>>
>>> Thanks,
>>>
>>> --
>>>
>>> *Tianxiang Xiong *| Clojure Engineer
>>>
>>> *tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
>>>
>>> 747 Front Street, Floor 4 | San Francisco, CA 94111
>>>
>>
>>
>
>
> --
>
> *Tianxiang Xiong *| Clojure Engineer
>
> *tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
>
> 747 Front Street, Floor 4 | San Francisco, CA 94111
>
Re: UUIDs in Avro Schemas
Posted by Tianxiang Xiong <ti...@fundingcircle.com>.
Thanks Yibing, I have created AVRO-1962
<https://issues.apache.org/jira/browse/AVRO-1962>. Take a look and let me
know if additional information/changes are needed to the ticket.
On 22 November 2016 at 15:12, Yibing Shi <ys...@cloudera.com> wrote:
> Hi Tianxiang,
>
> Looks like that the UUID type hasn't been added to IDL definition yet.
>
> https://github.com/apache/avro/blob/master/lang/java/
> compiler/src/main/javacc/org/apache/avro/compiler/idl/idl.jj#L214-L244
>
> Please create a JIRA to track this.
>
> *Yibing Shi*
> *Customer Operations Engineer*
> <http://www.cloudera.com>
>
> On Wed, Nov 23, 2016 at 4:58 AM, Tianxiang Xiong <tianxiang.xiong@
> fundingcircle.com> wrote:
>
>> The AVRO-1554 <https://issues.apache.org/jira/browse/AVRO-1554> ticket
>> seems to suggest that the issue of representing UUIDs is resolved with
>> logical
>> types <http://avro.apache.org/docs/1.8.1/spec.html#Logical+Types> in Avro
>> 1.8.1.
>>
>> The specification offers several examples of using logical types; decimals
>> are represented as:
>>
>> {
>>
>> "type": "bytes",
>> "logicalType": "decimal",
>> "precision": 4,
>> "scale": 2
>> }
>>
>> However, there does not seem to be an example of UUID. The most
>> popular answers <http://stackoverflow.com/a/19705149/864684> via
>> Google/StackOverflow are a few years old and do not make use of
>> logical types, and can therefore be misleading for newer versions of
>> Avro
>>
>> Is there an idiomatic way of representing UUIDs in Avro Schemas?
>>
>> Thanks,
>>
>> --
>>
>> *Tianxiang Xiong *| Clojure Engineer
>>
>> *tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
>>
>> 747 Front Street, Floor 4 | San Francisco, CA 94111
>>
>
>
--
*Tianxiang Xiong *| Clojure Engineer
*tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
747 Front Street, Floor 4 | San Francisco, CA 94111
Re: UUIDs in Avro Schemas
Posted by Yibing Shi <ys...@cloudera.com>.
Hi Tianxiang,
Looks like that the UUID type hasn't been added to IDL definition yet.
https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/javacc/org/apache/avro/compiler/idl/idl.jj#L214-L244
Please create a JIRA to track this.
*Yibing Shi*
*Customer Operations Engineer*
<http://www.cloudera.com>
On Wed, Nov 23, 2016 at 4:58 AM, Tianxiang Xiong <
tianxiang.xiong@fundingcircle.com> wrote:
> The AVRO-1554 <https://issues.apache.org/jira/browse/AVRO-1554> ticket
> seems to suggest that the issue of representing UUIDs is resolved with
> logical
> types <http://avro.apache.org/docs/1.8.1/spec.html#Logical+Types> in Avro
> 1.8.1.
>
> The specification offers several examples of using logical types; decimals
> are represented as:
>
> {
>
> "type": "bytes",
> "logicalType": "decimal",
> "precision": 4,
> "scale": 2
> }
>
> However, there does not seem to be an example of UUID. The most
> popular answers <http://stackoverflow.com/a/19705149/864684> via
> Google/StackOverflow are a few years old and do not make use of
> logical types, and can therefore be misleading for newer versions of
> Avro
>
> Is there an idiomatic way of representing UUIDs in Avro Schemas?
>
> Thanks,
>
> --
>
> *Tianxiang Xiong *| Clojure Engineer
>
> *tianxiang.xiong@fundingcircle.com <ti...@fundingcircle.com>*
>
> 747 Front Street, Floor 4 | San Francisco, CA 94111
>