optimal way to store historical data

[Kirill Lykov <ly...@gmail.com>]

Hi,

I wonder  what is the best way to represent time series in the arrow.
Maybe someone did a research already about different ways of
representing these data? Or there is a ready-to-use solution inside
the library. Basically, I need a third dimension to the table which is
time. One of the solutions is to have a new table for each date, but
there are many other ways as well.

-- 
Best regards,
Kirill Lykov

Re: optimal way to store historical data

[an...@gmail.com]

Also, let me clarify so there is no confusion  - There should be no problem
creating static / read only arrow data files with a 'date to batch' index
in the manner i described.  The problem I am referring to only becomes an
issue if you need to append a new batch on a daily basis

-Anthony


On Fri, Jun 26, 2020 at 11:58 AM <an...@gmail.com> wrote:

> +1 to this..
>
> There is a logical way to do this now - If you create a batch per day you
> can maintain a separate arrow file (an index) to map the date to batch.. We
> do this for indexing via other keys, and I can say it works well for
> 'large' files - 25gb+. I think unfortunately, doing this via the current
> language level API requires a lot of recreating a copy files for each new
> batch due to the immutability
>
> However, I know for a fact the file based format could support this 'very
> easily' it would just require the footer to be replaced and overwritten
> with the next batch and a new block inserted with correct offsets into a
> new footer after the new batch.   Using the C++ libraries (or your own) one
> could do this manually, but it might be useful to build it into the
> higher level APIs.  If there exists a way to do this now that doesnt
> require hacking the files manually, please let me know!
>
> On Fri, Jun 26, 2020 at 11:49 AM Dachuan Zhao <tc...@gmail.com>
> wrote:
>
>> +1
>> Is the dataset the model for that?
>>
>> On Fri, Jun 26, 2020 at 11:42 AM Kirill Lykov <ly...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I wonder  what is the best way to represent time series in the arrow.
>> > Maybe someone did a research already about different ways of
>> > representing these data? Or there is a ready-to-use solution inside
>> > the library. Basically, I need a third dimension to the table which is
>> > time. One of the solutions is to have a new table for each date, but
>> > there are many other ways as well.
>> >
>> > --
>> > Best regards,
>> > Kirill Lykov
>> >
>> --
>> Sent from iPhone
>>
>

Re: optimal way to store historical data

[an...@gmail.com]

+1 to this..

There is a logical way to do this now - If you create a batch per day you
can maintain a separate arrow file (an index) to map the date to batch.. We
do this for indexing via other keys, and I can say it works well for
'large' files - 25gb+. I think unfortunately, doing this via the current
language level API requires a lot of recreating a copy files for each new
batch due to the immutability

However, I know for a fact the file based format could support this 'very
easily' it would just require the footer to be replaced and overwritten
with the next batch and a new block inserted with correct offsets into a
new footer after the new batch.   Using the C++ libraries (or your own) one
could do this manually, but it might be useful to build it into the
higher level APIs.  If there exists a way to do this now that doesnt
require hacking the files manually, please let me know!

On Fri, Jun 26, 2020 at 11:49 AM Dachuan Zhao <tc...@gmail.com>
wrote:

> +1
> Is the dataset the model for that?
>
> On Fri, Jun 26, 2020 at 11:42 AM Kirill Lykov <ly...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I wonder  what is the best way to represent time series in the arrow.
> > Maybe someone did a research already about different ways of
> > representing these data? Or there is a ready-to-use solution inside
> > the library. Basically, I need a third dimension to the table which is
> > time. One of the solutions is to have a new table for each date, but
> > there are many other ways as well.
> >
> > --
> > Best regards,
> > Kirill Lykov
> >
> --
> Sent from iPhone
>

Re: optimal way to store historical data

[Dachuan Zhao <tc...@gmail.com>]

+1
Is the dataset the model for that?

On Fri, Jun 26, 2020 at 11:42 AM Kirill Lykov <ly...@gmail.com>
wrote:

> Hi,
>
> I wonder  what is the best way to represent time series in the arrow.
> Maybe someone did a research already about different ways of
> representing these data? Or there is a ready-to-use solution inside
> the library. Basically, I need a third dimension to the table which is
> time. One of the solutions is to have a new table for each date, but
> there are many other ways as well.
>
> --
> Best regards,
> Kirill Lykov
>
-- 
Sent from iPhone