You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Lewis John Mcgibbney <le...@gmail.com> on 2014/05/06 00:54:28 UTC
Dynamic Package/namespace naming
Hi Folks,
I'm trying to propose Afro for a project I've been drafted on to.
Namespace declarations are important... and a requirement is that
namespaces are dynamic in nature... preferably even decided at runtime.
Is this possible in Avro?
Thanks
Lewis
Re: Dynamic Package/namespace naming
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Sean thanks for the additional info. This is really helpful.
Some of these constraints are not what I would prefer... and of course mean
more work. Powers that be request them ;)
Thanks
On May 6, 2014 9:06 AM, "Sean Busbey" <bu...@cloudera.com> wrote:
> Hi Lewis,
>
> Well, I'm not sure what you're going to gain by isolating each file into
> its own namespace.
>
> However, one way you can achieve that is to take the information you want
> included from before and then create a hash of it. You can use the hex
> representation of the hash to give you something that
>
> * meets Avro's requirements for namespace names
> * uniquely identifies each namespace based on experiment, channel, and
> timestamp
>
> If you need the file to also contain the information that went into making
> that hash, you can attach it as Avro-ignored metadata on the schema.
>
> -Sean
>
>
> On Mon, May 5, 2014 at 6:49 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Hi Sean,
>> Arggggh...
>> Maybe I can propose a change to the namespace semantics in my project.
>> I've been given the following brief.
>> Data files should have a logical namespace structure which can be
>> globally unified across multiple instruments and receiver platforms into a
>> common namespace on the filesystem.
>> I have a whole list of other data representation constraints and Avro
>> satisfies them all. This one is the only 'issue'.
>> Thanks for your help.
>> Lewis
>> On May 5, 2014 4:21 PM, "Sean Busbey" <bu...@cloudera.com> wrote:
>>
>>> Hi Lewis!
>>>
>>> Avro namespaces don't allow the characters '/', ':', or '-'. So your
>>> specific example would not work. The allowed characters for a namespace
>>> are defined in the Avro spec[1].
>>>
>>> It would help if you could clarify what purpose namespacing serves in
>>> the system.
>>>
>>> [1]: http://avro.apache.org/docs/current/spec.html#Names
>>>
>>>
>>> On Mon, May 5, 2014 at 6:14 PM, Lewis John Mcgibbney <
>>> lewis.mcgibbney@gmail.com> wrote:
>>>
>>>> Hi Charles
>>>> Thanks for reply.
>>>> Re: more info
>>>> Say I want my namespace to be
>>>>
>>>> experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/
>>>>
>>>> Where all constituent parts of namespace separated by '/' are
>>>> dynamic... is this possible via builder api?
>>>> Thanks again
>>>> On May 5, 2014 4:04 PM, "Pritchard, Charles X. -ND" <
>>>> Charles.X.Pritchard.-ND@disney.com> wrote:
>>>>
>>>>> Need more info — you can use the Schema builder to do anything you
>>>>> like at runtime,
>>>>> Schema.createRecord and setFields.
>>>>>
>>>>>
>>>>>
>>>>> On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <
>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>
>>>>> Hi Folks,
>>>>> I'm trying to propose Afro for a project I've been drafted on to.
>>>>> Namespace declarations are important... and a requirement is that
>>>>> namespaces are dynamic in nature... preferably even decided at runtime.
>>>>> Is this possible in Avro?
>>>>> Thanks
>>>>> Lewis
>>>>>
>>>>>
>>>>>
>>>
>
>
> --
> Sean
>
Re: Dynamic Package/namespace naming
Posted by Sean Busbey <bu...@cloudera.com>.
Hi Lewis,
Well, I'm not sure what you're going to gain by isolating each file into
its own namespace.
However, one way you can achieve that is to take the information you want
included from before and then create a hash of it. You can use the hex
representation of the hash to give you something that
* meets Avro's requirements for namespace names
* uniquely identifies each namespace based on experiment, channel, and
timestamp
If you need the file to also contain the information that went into making
that hash, you can attach it as Avro-ignored metadata on the schema.
-Sean
On Mon, May 5, 2014 at 6:49 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi Sean,
> Arggggh...
> Maybe I can propose a change to the namespace semantics in my project.
> I've been given the following brief.
> Data files should have a logical namespace structure which can be globally
> unified across multiple instruments and receiver platforms into a common
> namespace on the filesystem.
> I have a whole list of other data representation constraints and Avro
> satisfies them all. This one is the only 'issue'.
> Thanks for your help.
> Lewis
> On May 5, 2014 4:21 PM, "Sean Busbey" <bu...@cloudera.com> wrote:
>
>> Hi Lewis!
>>
>> Avro namespaces don't allow the characters '/', ':', or '-'. So your
>> specific example would not work. The allowed characters for a namespace
>> are defined in the Avro spec[1].
>>
>> It would help if you could clarify what purpose namespacing serves in the
>> system.
>>
>> [1]: http://avro.apache.org/docs/current/spec.html#Names
>>
>>
>> On Mon, May 5, 2014 at 6:14 PM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>
>>> Hi Charles
>>> Thanks for reply.
>>> Re: more info
>>> Say I want my namespace to be
>>>
>>> experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/
>>>
>>> Where all constituent parts of namespace separated by '/' are dynamic...
>>> is this possible via builder api?
>>> Thanks again
>>> On May 5, 2014 4:04 PM, "Pritchard, Charles X. -ND" <
>>> Charles.X.Pritchard.-ND@disney.com> wrote:
>>>
>>>> Need more info — you can use the Schema builder to do anything you like
>>>> at runtime,
>>>> Schema.createRecord and setFields.
>>>>
>>>>
>>>>
>>>> On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <
>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>
>>>> Hi Folks,
>>>> I'm trying to propose Afro for a project I've been drafted on to.
>>>> Namespace declarations are important... and a requirement is that
>>>> namespaces are dynamic in nature... preferably even decided at runtime.
>>>> Is this possible in Avro?
>>>> Thanks
>>>> Lewis
>>>>
>>>>
>>>>
>>
--
Sean
Re: Dynamic Package/namespace naming
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Sean,
Arggggh...
Maybe I can propose a change to the namespace semantics in my project.
I've been given the following brief.
Data files should have a logical namespace structure which can be globally
unified across multiple instruments and receiver platforms into a common
namespace on the filesystem.
I have a whole list of other data representation constraints and Avro
satisfies them all. This one is the only 'issue'.
Thanks for your help.
Lewis
On May 5, 2014 4:21 PM, "Sean Busbey" <bu...@cloudera.com> wrote:
> Hi Lewis!
>
> Avro namespaces don't allow the characters '/', ':', or '-'. So your
> specific example would not work. The allowed characters for a namespace
> are defined in the Avro spec[1].
>
> It would help if you could clarify what purpose namespacing serves in the
> system.
>
> [1]: http://avro.apache.org/docs/current/spec.html#Names
>
>
> On Mon, May 5, 2014 at 6:14 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Hi Charles
>> Thanks for reply.
>> Re: more info
>> Say I want my namespace to be
>>
>> experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/
>>
>> Where all constituent parts of namespace separated by '/' are dynamic...
>> is this possible via builder api?
>> Thanks again
>> On May 5, 2014 4:04 PM, "Pritchard, Charles X. -ND" <
>> Charles.X.Pritchard.-ND@disney.com> wrote:
>>
>>> Need more info — you can use the Schema builder to do anything you like
>>> at runtime,
>>> Schema.createRecord and setFields.
>>>
>>>
>>>
>>> On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <
>>> lewis.mcgibbney@gmail.com> wrote:
>>>
>>> Hi Folks,
>>> I'm trying to propose Afro for a project I've been drafted on to.
>>> Namespace declarations are important... and a requirement is that
>>> namespaces are dynamic in nature... preferably even decided at runtime.
>>> Is this possible in Avro?
>>> Thanks
>>> Lewis
>>>
>>>
>>>
>
Re: Dynamic Package/namespace naming
Posted by Sean Busbey <bu...@cloudera.com>.
Hi Lewis!
Avro namespaces don't allow the characters '/', ':', or '-'. So your
specific example would not work. The allowed characters for a namespace
are defined in the Avro spec[1].
It would help if you could clarify what purpose namespacing serves in the
system.
[1]: http://avro.apache.org/docs/current/spec.html#Names
On Mon, May 5, 2014 at 6:14 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi Charles
> Thanks for reply.
> Re: more info
> Say I want my namespace to be
>
> experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/
>
> Where all constituent parts of namespace separated by '/' are dynamic...
> is this possible via builder api?
> Thanks again
> On May 5, 2014 4:04 PM, "Pritchard, Charles X. -ND" <
> Charles.X.Pritchard.-ND@disney.com> wrote:
>
>> Need more info — you can use the Schema builder to do anything you like
>> at runtime,
>> Schema.createRecord and setFields.
>>
>>
>>
>> On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>
>> Hi Folks,
>> I'm trying to propose Afro for a project I've been drafted on to.
>> Namespace declarations are important... and a requirement is that
>> namespaces are dynamic in nature... preferably even decided at runtime.
>> Is this possible in Avro?
>> Thanks
>> Lewis
>>
>>
>>
Re: Dynamic Package/namespace naming
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Charles
Thanks for reply.
Re: more info
Say I want my namespace to be
experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/
Where all constituent parts of namespace separated by '/' are dynamic... is
this possible via builder api?
Thanks again
On May 5, 2014 4:04 PM, "Pritchard, Charles X. -ND" <
Charles.X.Pritchard.-ND@disney.com> wrote:
> Need more info — you can use the Schema builder to do anything you like at
> runtime,
> Schema.createRecord and setFields.
>
>
>
> On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> Hi Folks,
> I'm trying to propose Afro for a project I've been drafted on to.
> Namespace declarations are important... and a requirement is that
> namespaces are dynamic in nature... preferably even decided at runtime.
> Is this possible in Avro?
> Thanks
> Lewis
>
>
>
Re: Dynamic Package/namespace naming
Posted by "Pritchard, Charles X. -ND" <Ch...@disney.com>.
Need more info — you can use the Schema builder to do anything you like at runtime,
Schema.createRecord and setFields.
On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <le...@gmail.com>> wrote:
Hi Folks,
I'm trying to propose Afro for a project I've been drafted on to.
Namespace declarations are important... and a requirement is that namespaces are dynamic in nature... preferably even decided at runtime.
Is this possible in Avro?
Thanks
Lewis