Efficient storage of metadata

[Robert McLeod <ro...@gmail.com>]

What is considered best practice for storage of fairly large quantities of
meta-data within Arrow? For the sake of simplicity I'll define metadata as
key-value pairs, in this case stored in a 2-column SQL table, but it
equally could be a Python dict or similar.

Arrow has a dictionary class but it doesn't appear to permit the type of
the value to be mutable. Since it seems to be some sort of std map I
discounted it as a solution, since I have int, float, and string values.

My current thought would be to serialize the data to JSON and write it into
a `ListArray` with bytes type, with one element in the ListArray for every
JSON blob I have. It feels a bit kludge but it does work. Is this a
reasonable approach, or is there a better solution?

Robert


-- 
Robert McLeod
robbmcleod@gmail.com
robert.mcleod@hitachi-hightech.com

Re: Efficient storage of metadata

[Weston Pace <we...@gmail.com>]

Spencer's answer is good.  I'll just add one minor clarification.  A
MapArray is what you want, but it is a List<Struct<Key, Value>> and not
just Struct<Key, Value>.  This allows you to have many rows of metadata
key/pairs for each row of your structured data.

On Thu, May 18, 2023 at 5:41 PM Spencer Nelson <sw...@uw.edu> wrote:

> You could use a struct with two fields: a string key, and a string value
> (or union-typed value if you want multiple legal types).
>
> That’s what the pyarrow MapArray type is, under the hood; if you’re in
> Python then using map would be simple and convenient.
>
> On Thu, May 18, 2023 at 5:07 PM Robert McLeod <ro...@gmail.com>
> wrote:
>
>> What is considered best practice for storage of fairly large quantities
>> of meta-data within Arrow? For the sake of simplicity I'll define metadata
>> as key-value pairs, in this case stored in a 2-column SQL table, but it
>> equally could be
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an Untrusted Sender
>> You have not previously corresponded with this sender.
>> See https://itconnect.uw.edu/email-tags for additional information.
>> Please contact the UW-IT Service Center, help@uw.edu 206.221.5000, for
>> assistance.
>>
>> ZjQcmQRYFpfptBannerEnd
>> What is considered best practice for storage of fairly large quantities
>> of meta-data within Arrow? For the sake of simplicity I'll define metadata
>> as key-value pairs, in this case stored in a 2-column SQL table, but it
>> equally could be a Python dict or similar.
>>
>> Arrow has a dictionary class but it doesn't appear to permit the type of
>> the value to be mutable. Since it seems to be some sort of std map I
>> discounted it as a solution, since I have int, float, and string values.
>>
>> My current thought would be to serialize the data to JSON and write it
>> into a `ListArray` with bytes type, with one element in the ListArray for
>> every JSON blob I have. It feels a bit kludge but it does work. Is this a
>> reasonable approach, or is there a better solution?
>>
>> Robert
>>
>>
>> --
>> Robert McLeod
>> robbmcleod@gmail.com
>> robert.mcleod@hitachi-hightech.com
>>
>>

Re: Efficient storage of metadata

[Spencer Nelson <sw...@uw.edu>]

You could use a struct with two fields: a string key, and a string value
(or union-typed value if you want multiple legal types).

That’s what the pyarrow MapArray type is, under the hood; if you’re in
Python then using map would be simple and convenient.

On Thu, May 18, 2023 at 5:07 PM Robert McLeod <ro...@gmail.com> wrote:

> What is considered best practice for storage of fairly large quantities of
> meta-data within Arrow? For the sake of simplicity I'll define metadata as
> key-value pairs, in this case stored in a 2-column SQL table, but it
> equally could be
> ZjQcmQRYFpfptBannerStart
> This Message Is From an Untrusted Sender
> You have not previously corresponded with this sender.
> See https://itconnect.uw.edu/email-tags for additional information.
> Please contact the UW-IT Service Center, help@uw.edu 206.221.5000, for
> assistance.
>
> ZjQcmQRYFpfptBannerEnd
> What is considered best practice for storage of fairly large quantities of
> meta-data within Arrow? For the sake of simplicity I'll define metadata as
> key-value pairs, in this case stored in a 2-column SQL table, but it
> equally could be a Python dict or similar.
>
> Arrow has a dictionary class but it doesn't appear to permit the type of
> the value to be mutable. Since it seems to be some sort of std map I
> discounted it as a solution, since I have int, float, and string values.
>
> My current thought would be to serialize the data to JSON and write it
> into a `ListArray` with bytes type, with one element in the ListArray for
> every JSON blob I have. It feels a bit kludge but it does work. Is this a
> reasonable approach, or is there a better solution?
>
> Robert
>
>
> --
> Robert McLeod
> robbmcleod@gmail.com
> robert.mcleod@hitachi-hightech.com
>
>