You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by gordon chung <go...@live.ca> on 2021/11/03 19:00:52 UTC
feather file and arrow internals
hi,
apologies if this in the doc or mailing list somewhere and I missed it but I was hoping to understand the arrow file format a bit more.
I noticed that when reading a feather file, the API, at least for Python, doesn't support filtering. is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api? there are references that it supports slicing but I'm thinking more like filtering to only get rows fitting a specific condition (get rows where col1 == 'a' vs get rows 1,3,5...).
also, most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet. that said, the featherv2 post does have comparison against parquet and my limited testing does show featherv2 performing favourably against it. i guess the question is, should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
thanks,
gord
Re: feather file and arrow internals
Posted by gordon chung <go...@live.ca>.
> This is done automatically by
> pyarrow.dataset.Dataset.to_table and
> pyarrow.dataset.Dataset.to_batches if you provide a filter. > The datasets API will first try and use the file format to reduce the
> amount of data it has to load (predicate pushdown). If it cannot do
> this (e.g. with the feather format), then it will still apply the
> filter in a streaming / out-of-core fashion.
nice! was going to ask exactly this, was wondering how datasets had filter option but feather support. the dataset docs list the API as experimental, is there timeline to finalise this? I imagine it's not a huge deal as the underlying storage options are stable but just out of curiosity.
thanks again for all the info.
cheers,
gord
________________________________
From: Weston Pace <we...@gmail.com>
Sent: Thursday, November 4, 2021 1:25:46 AM
To: user@arrow.apache.org <us...@arrow.apache.org>
Subject: Re: feather file and arrow internals
> interesting. just curious, but is there a way to do it without extending the format? like it would still scan through all the batches but it wouldn’t load all rows into memory or maybe it’s already possible? for example, if i write a 1GB arrow table as a feather file and it’s 1GB uncompressed on disk. can i read_table on that file but apply filtering as it scans so the resulting table is never 1GB? not sure if this conflicts with zero-copy and/or SIMD design.
That is definitely possible today. In pyarrow you can get this
functionality with the datasets API and it doesn't matter if you use
parquet or feather. This is done automatically by
pyarrow.dataset.Dataset.to_table and
pyarrow.dataset.Dataset.to_batches if you provide a filter. The
datasets API will first try and use the file format to reduce the
amount of data it has to load (predicate pushdown). If it cannot do
this (e.g. with the feather format), then it will still apply the
filter in a streaming / out-of-core fashion.
One important note is that your feather file will need to have more
than 1 row group for this to work. If you have a 100GB feather file
stored as a single record batch and you try to apply a filter to it
then it will occupy 100GB of ram (note: this is a limitation of the
datasets API / IPC reader and not the feather file format as the
feather file format supports random slicing). If you have a 100GB
feather file stored as 1,000 100MB record batches then it will never
occupy 100GB of RAM. Maybe size your record batches so that you can
fit about 100 of them into RAM (note: how many actual batches are
stored in memory is a factor of how many files you have, how many
record batches each file has, your file format, your fragment
readahead parameter, and the batch readahead parameter. These last
two parameters are not exposed in pyarrow today). If you only have
one file it shouldn't load more than ~32 batches into RAM at once.
Finally, the resulting table (if you are running to_table) will not be
1GB. The filter passed to the `to_table` function is applied as the
table is constructed.
On Wed, Nov 3, 2021 at 12:30 PM gordon chung <go...@live.ca> wrote:
>
> thanks for the details! really helpful.
>
> > However, there is a spot for it (record batch metadata) and there has
> > been discussion in the past of adding similar capabilities someday.
>
> interesting. just curious, but is there a way to do it without extending the format? like it would still scan through all the batches but it wouldn’t load all rows into memory or maybe it’s already possible? for example, if i write a 1GB arrow table as a feather file and it’s 1GB uncompressed on disk. can i read_table on that file but apply filtering as it scans so the resulting table is never 1GB? not sure if this conflicts with zero-copy and/or SIMD design.
>
> > As a general rule of thumb parquet is more space-efficient
> > and should be used when you are limited by I/O bandwidth. Feather is
> > more CPU-efficient and should be used when you are limited by CPU
> > bandwidth.
>
> this was really useful way of describing when to choose which at a high-level. i’ll definitely need to take a closer look at feather now to test this out.
>
> ________________________________________
> From: Weston Pace <we...@gmail.com>
> Sent: November 3, 2021 16:54
> To: user@arrow.apache.org
> Subject: Re: feather file and arrow internals
>
> Great questions.
>
> > is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api?
>
> The former. Row filtering in parquet relies on row group statistics
> (min & max values) and someday we may also support using bloom filters
> (more than just min/max) and data page statistics (still min/max but
> at a finer resolution). The feather-v2 format (a.k.a Arrow-IPC) does
> not have any defined standard for storing row group statistics.
> However, there is a spot for it (record batch metadata) and there has
> been discussion in the past of adding similar capabilities someday.
> If someone had enough motivation I think all the necessary parts are
> ready so it is mainly just waiting for someone with motivation and
> engineering time.
>
> > should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
>
> Feather-v1 is something of a proof of concept (although we are
> maintaining backwards compatibility with it). Feather-v2, which is
> sometimes just called the Arrow IPC format, is definitely intended to
> be maintained and not just a proof of concept.
>
> > most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet.
>
> Feather and parquet have different use cases and it's difficult to
> describe which is more appropriate as it can depend on a lot of
> details. As a general rule of thumb parquet is more space-efficient
> and should be used when you are limited by I/O bandwidth. Feather is
> more CPU-efficient and should be used when you are limited by CPU
> bandwidth. However, this is only a rule of thumb and there are plenty
> of exceptions.
>
> On Wed, Nov 3, 2021 at 9:01 AM gordon chung <go...@live.ca> wrote:
> >
> > hi,
> >
> > apologies if this in the doc or mailing list somewhere and I missed it but I was hoping to understand the arrow file format a bit more.
> >
> > I noticed that when reading a feather file, the API, at least for Python, doesn't support filtering. is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api? there are references that it supports slicing but I'm thinking more like filtering to only get rows fitting a specific condition (get rows where col1 == 'a' vs get rows 1,3,5...).
> >
> > also, most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet. that said, the featherv2 post does have comparison against parquet and my limited testing does show featherv2 performing favourably against it. i guess the question is, should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
> >
> > thanks,
> >
> > gord
Re: feather file and arrow internals
Posted by Weston Pace <we...@gmail.com>.
> interesting. just curious, but is there a way to do it without extending the format? like it would still scan through all the batches but it wouldn’t load all rows into memory or maybe it’s already possible? for example, if i write a 1GB arrow table as a feather file and it’s 1GB uncompressed on disk. can i read_table on that file but apply filtering as it scans so the resulting table is never 1GB? not sure if this conflicts with zero-copy and/or SIMD design.
That is definitely possible today. In pyarrow you can get this
functionality with the datasets API and it doesn't matter if you use
parquet or feather. This is done automatically by
pyarrow.dataset.Dataset.to_table and
pyarrow.dataset.Dataset.to_batches if you provide a filter. The
datasets API will first try and use the file format to reduce the
amount of data it has to load (predicate pushdown). If it cannot do
this (e.g. with the feather format), then it will still apply the
filter in a streaming / out-of-core fashion.
One important note is that your feather file will need to have more
than 1 row group for this to work. If you have a 100GB feather file
stored as a single record batch and you try to apply a filter to it
then it will occupy 100GB of ram (note: this is a limitation of the
datasets API / IPC reader and not the feather file format as the
feather file format supports random slicing). If you have a 100GB
feather file stored as 1,000 100MB record batches then it will never
occupy 100GB of RAM. Maybe size your record batches so that you can
fit about 100 of them into RAM (note: how many actual batches are
stored in memory is a factor of how many files you have, how many
record batches each file has, your file format, your fragment
readahead parameter, and the batch readahead parameter. These last
two parameters are not exposed in pyarrow today). If you only have
one file it shouldn't load more than ~32 batches into RAM at once.
Finally, the resulting table (if you are running to_table) will not be
1GB. The filter passed to the `to_table` function is applied as the
table is constructed.
On Wed, Nov 3, 2021 at 12:30 PM gordon chung <go...@live.ca> wrote:
>
> thanks for the details! really helpful.
>
> > However, there is a spot for it (record batch metadata) and there has
> > been discussion in the past of adding similar capabilities someday.
>
> interesting. just curious, but is there a way to do it without extending the format? like it would still scan through all the batches but it wouldn’t load all rows into memory or maybe it’s already possible? for example, if i write a 1GB arrow table as a feather file and it’s 1GB uncompressed on disk. can i read_table on that file but apply filtering as it scans so the resulting table is never 1GB? not sure if this conflicts with zero-copy and/or SIMD design.
>
> > As a general rule of thumb parquet is more space-efficient
> > and should be used when you are limited by I/O bandwidth. Feather is
> > more CPU-efficient and should be used when you are limited by CPU
> > bandwidth.
>
> this was really useful way of describing when to choose which at a high-level. i’ll definitely need to take a closer look at feather now to test this out.
>
> ________________________________________
> From: Weston Pace <we...@gmail.com>
> Sent: November 3, 2021 16:54
> To: user@arrow.apache.org
> Subject: Re: feather file and arrow internals
>
> Great questions.
>
> > is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api?
>
> The former. Row filtering in parquet relies on row group statistics
> (min & max values) and someday we may also support using bloom filters
> (more than just min/max) and data page statistics (still min/max but
> at a finer resolution). The feather-v2 format (a.k.a Arrow-IPC) does
> not have any defined standard for storing row group statistics.
> However, there is a spot for it (record batch metadata) and there has
> been discussion in the past of adding similar capabilities someday.
> If someone had enough motivation I think all the necessary parts are
> ready so it is mainly just waiting for someone with motivation and
> engineering time.
>
> > should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
>
> Feather-v1 is something of a proof of concept (although we are
> maintaining backwards compatibility with it). Feather-v2, which is
> sometimes just called the Arrow IPC format, is definitely intended to
> be maintained and not just a proof of concept.
>
> > most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet.
>
> Feather and parquet have different use cases and it's difficult to
> describe which is more appropriate as it can depend on a lot of
> details. As a general rule of thumb parquet is more space-efficient
> and should be used when you are limited by I/O bandwidth. Feather is
> more CPU-efficient and should be used when you are limited by CPU
> bandwidth. However, this is only a rule of thumb and there are plenty
> of exceptions.
>
> On Wed, Nov 3, 2021 at 9:01 AM gordon chung <go...@live.ca> wrote:
> >
> > hi,
> >
> > apologies if this in the doc or mailing list somewhere and I missed it but I was hoping to understand the arrow file format a bit more.
> >
> > I noticed that when reading a feather file, the API, at least for Python, doesn't support filtering. is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api? there are references that it supports slicing but I'm thinking more like filtering to only get rows fitting a specific condition (get rows where col1 == 'a' vs get rows 1,3,5...).
> >
> > also, most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet. that said, the featherv2 post does have comparison against parquet and my limited testing does show featherv2 performing favourably against it. i guess the question is, should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
> >
> > thanks,
> >
> > gord
Re: feather file and arrow internals
Posted by gordon chung <go...@live.ca>.
thanks for the details! really helpful.
> However, there is a spot for it (record batch metadata) and there has
> been discussion in the past of adding similar capabilities someday.
interesting. just curious, but is there a way to do it without extending the format? like it would still scan through all the batches but it wouldn’t load all rows into memory or maybe it’s already possible? for example, if i write a 1GB arrow table as a feather file and it’s 1GB uncompressed on disk. can i read_table on that file but apply filtering as it scans so the resulting table is never 1GB? not sure if this conflicts with zero-copy and/or SIMD design.
> As a general rule of thumb parquet is more space-efficient
> and should be used when you are limited by I/O bandwidth. Feather is
> more CPU-efficient and should be used when you are limited by CPU
> bandwidth.
this was really useful way of describing when to choose which at a high-level. i’ll definitely need to take a closer look at feather now to test this out.
________________________________________
From: Weston Pace <we...@gmail.com>
Sent: November 3, 2021 16:54
To: user@arrow.apache.org
Subject: Re: feather file and arrow internals
Great questions.
> is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api?
The former. Row filtering in parquet relies on row group statistics
(min & max values) and someday we may also support using bloom filters
(more than just min/max) and data page statistics (still min/max but
at a finer resolution). The feather-v2 format (a.k.a Arrow-IPC) does
not have any defined standard for storing row group statistics.
However, there is a spot for it (record batch metadata) and there has
been discussion in the past of adding similar capabilities someday.
If someone had enough motivation I think all the necessary parts are
ready so it is mainly just waiting for someone with motivation and
engineering time.
> should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
Feather-v1 is something of a proof of concept (although we are
maintaining backwards compatibility with it). Feather-v2, which is
sometimes just called the Arrow IPC format, is definitely intended to
be maintained and not just a proof of concept.
> most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet.
Feather and parquet have different use cases and it's difficult to
describe which is more appropriate as it can depend on a lot of
details. As a general rule of thumb parquet is more space-efficient
and should be used when you are limited by I/O bandwidth. Feather is
more CPU-efficient and should be used when you are limited by CPU
bandwidth. However, this is only a rule of thumb and there are plenty
of exceptions.
On Wed, Nov 3, 2021 at 9:01 AM gordon chung <go...@live.ca> wrote:
>
> hi,
>
> apologies if this in the doc or mailing list somewhere and I missed it but I was hoping to understand the arrow file format a bit more.
>
> I noticed that when reading a feather file, the API, at least for Python, doesn't support filtering. is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api? there are references that it supports slicing but I'm thinking more like filtering to only get rows fitting a specific condition (get rows where col1 == 'a' vs get rows 1,3,5...).
>
> also, most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet. that said, the featherv2 post does have comparison against parquet and my limited testing does show featherv2 performing favourably against it. i guess the question is, should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
>
> thanks,
>
> gord
Re: feather file and arrow internals
Posted by Weston Pace <we...@gmail.com>.
Sorry to double-post but I should also add one more difference.
Feather supports random access (either a single index or a slice) and
parquet doesn't. That's another big factor in choosing which format
you want to go with.
On Wed, Nov 3, 2021 at 10:54 AM Weston Pace <we...@gmail.com> wrote:
>
> Great questions.
>
> > is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api?
>
> The former. Row filtering in parquet relies on row group statistics
> (min & max values) and someday we may also support using bloom filters
> (more than just min/max) and data page statistics (still min/max but
> at a finer resolution). The feather-v2 format (a.k.a Arrow-IPC) does
> not have any defined standard for storing row group statistics.
> However, there is a spot for it (record batch metadata) and there has
> been discussion in the past of adding similar capabilities someday.
> If someone had enough motivation I think all the necessary parts are
> ready so it is mainly just waiting for someone with motivation and
> engineering time.
>
> > should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
>
> Feather-v1 is something of a proof of concept (although we are
> maintaining backwards compatibility with it). Feather-v2, which is
> sometimes just called the Arrow IPC format, is definitely intended to
> be maintained and not just a proof of concept.
>
> > most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet.
>
> Feather and parquet have different use cases and it's difficult to
> describe which is more appropriate as it can depend on a lot of
> details. As a general rule of thumb parquet is more space-efficient
> and should be used when you are limited by I/O bandwidth. Feather is
> more CPU-efficient and should be used when you are limited by CPU
> bandwidth. However, this is only a rule of thumb and there are plenty
> of exceptions.
>
> On Wed, Nov 3, 2021 at 9:01 AM gordon chung <go...@live.ca> wrote:
> >
> > hi,
> >
> > apologies if this in the doc or mailing list somewhere and I missed it but I was hoping to understand the arrow file format a bit more.
> >
> > I noticed that when reading a feather file, the API, at least for Python, doesn't support filtering. is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api? there are references that it supports slicing but I'm thinking more like filtering to only get rows fitting a specific condition (get rows where col1 == 'a' vs get rows 1,3,5...).
> >
> > also, most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet. that said, the featherv2 post does have comparison against parquet and my limited testing does show featherv2 performing favourably against it. i guess the question is, should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
> >
> > thanks,
> >
> > gord
Re: feather file and arrow internals
Posted by Weston Pace <we...@gmail.com>.
Great questions.
> is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api?
The former. Row filtering in parquet relies on row group statistics
(min & max values) and someday we may also support using bloom filters
(more than just min/max) and data page statistics (still min/max but
at a finer resolution). The feather-v2 format (a.k.a Arrow-IPC) does
not have any defined standard for storing row group statistics.
However, there is a spot for it (record batch metadata) and there has
been discussion in the past of adding similar capabilities someday.
If someone had enough motivation I think all the necessary parts are
ready so it is mainly just waiting for someone with motivation and
engineering time.
> should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
Feather-v1 is something of a proof of concept (although we are
maintaining backwards compatibility with it). Feather-v2, which is
sometimes just called the Arrow IPC format, is definitely intended to
be maintained and not just a proof of concept.
> most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet.
Feather and parquet have different use cases and it's difficult to
describe which is more appropriate as it can depend on a lot of
details. As a general rule of thumb parquet is more space-efficient
and should be used when you are limited by I/O bandwidth. Feather is
more CPU-efficient and should be used when you are limited by CPU
bandwidth. However, this is only a rule of thumb and there are plenty
of exceptions.
On Wed, Nov 3, 2021 at 9:01 AM gordon chung <go...@live.ca> wrote:
>
> hi,
>
> apologies if this in the doc or mailing list somewhere and I missed it but I was hoping to understand the arrow file format a bit more.
>
> I noticed that when reading a feather file, the API, at least for Python, doesn't support filtering. is this because internally there is no metadata as to what a RecordBatch contains and it has to iterate through all batches or it is just something unsupported by api? there are references that it supports slicing but I'm thinking more like filtering to only get rows fitting a specific condition (get rows where col1 == 'a' vs get rows 1,3,5...).
>
> also, most references to feather/storing arrow on disk have historically had a disclaimer saying it's not meant to replace parquet. that said, the featherv2 post does have comparison against parquet and my limited testing does show featherv2 performing favourably against it. i guess the question is, should I use featherv2 in production if I'm ok with "drawbacks" (larger file, less adoption, other stuff I'm not aware of...) or is feather just a poc?
>
> thanks,
>
> gord