You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Sourav Mazumder <so...@gmail.com> on 2016/02/25 17:10:22 UTC
Comparing with Parquet
Hi All,
New to this. And still trying to figure out where exactly Arrow fits in the
ecosystem of various Big Data technologies.
In that respect first thing which came to my mind is how does Arrow compare
with parquet.
In my understanding Parquet also supports a very efficient columnar format
(with support for nested structure). It is already embraced (supported) by
various technologies like Impala (origin), Spark, Drill etc.
The only think I see missing in Parquet is support for SIMD based
vectorized operations.
Am I right or am I missing many other differences between Arrow and parquet
?
Regards,
Sourav
Re: Comparing with Parquet
Posted by Venkat Krishnamurthy <ni...@gmail.com>.
I think they're fundamentally orthogonal.
Tachyon provides a full-fledged storage system that uses memory as one of
its tiers, which leaves it up to applications as to how they represent
their data structures. When applications want to use Tachyon, it's up to
them to decide how to serialize their data structures.
Arrow, OTOH, leaves it up to the applications as to how they manage
storage. It is an in-memory data layout that exploits vectorization, and
(today) provides a library for defining data structures used in analytics
frameworks that need to be shared ideally with minimal overhead.
On Thu, Feb 25, 2016 at 5:11 PM, Pedro Miguel Duarte <
duarte.gelvez.pedromiguel@gmail.com> wrote:
> I was wondering if someone could also elaborate in the comparison with
> Tachyon (now called Alluxio)
> On Feb 25, 2016 5:08 PM, "Chenliang (Liang, DataSight)" <
> chenliang613@huawei.com> wrote:
>
> > In favor of Henry Robinson's points.
> >
> > In addition. Arrow is suitable for exchanging data high efficiently, but
> > the data size may just support TB level. Parquet can support more bigger
> > data, but the performance couldn't support fast query.
> >
> > So for PB level data and interactively query(second level), both couldn't
> > solve?
> >
> > Regards
> > Liang
> > -----邮件原件-----
> > 发件人: Henry Robinson [mailto:henry@cloudera.com]
> > 发送时间: 2016年2月26日 0:20
> > 收件人: dev@arrow.apache.org
> > 主题: Re: Comparing with Parquet
> >
> > Think of Parquet as a format well-suited to writing very large datasets
> to
> > disk, whereas Arrow is a format most suited to efficient storage in
> memory.
> > You might read Parquet files from disk, and then materialize them in
> memory
> > in Arrow's format.
> >
> > Both formats are designed around the idiosyncrasies of the target medium:
> > Parquet is not designed to support efficient random access because disks
> > aren't good at that, but Arrow has fast random access as a core design
> > principle, to give just one example.
> >
> > Henry
> >
> > > On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <
> > sourav.mazumder00@gmail.com> wrote:
> > >
> > > Hi All,
> > >
> > > New to this. And still trying to figure out where exactly Arrow fits
> > > in the ecosystem of various Big Data technologies.
> > >
> > > In that respect first thing which came to my mind is how does Arrow
> > > compare with parquet.
> > >
> > > In my understanding Parquet also supports a very efficient columnar
> > > format (with support for nested structure). It is already embraced
> > > (supported) by various technologies like Impala (origin), Spark, Drill
> > etc.
> > >
> > > The only think I see missing in Parquet is support for SIMD based
> > > vectorized operations.
> > >
> > > Am I right or am I missing many other differences between Arrow and
> > > parquet ?
> > >
> > > Regards,
> > > Sourav
> >
>
Re: Comparing with Parquet
Posted by Pedro Miguel Duarte <du...@gmail.com>.
I was wondering if someone could also elaborate in the comparison with
Tachyon (now called Alluxio)
On Feb 25, 2016 5:08 PM, "Chenliang (Liang, DataSight)" <
chenliang613@huawei.com> wrote:
> In favor of Henry Robinson's points.
>
> In addition. Arrow is suitable for exchanging data high efficiently, but
> the data size may just support TB level. Parquet can support more bigger
> data, but the performance couldn't support fast query.
>
> So for PB level data and interactively query(second level), both couldn't
> solve?
>
> Regards
> Liang
> -----邮件原件-----
> 发件人: Henry Robinson [mailto:henry@cloudera.com]
> 发送时间: 2016年2月26日 0:20
> 收件人: dev@arrow.apache.org
> 主题: Re: Comparing with Parquet
>
> Think of Parquet as a format well-suited to writing very large datasets to
> disk, whereas Arrow is a format most suited to efficient storage in memory.
> You might read Parquet files from disk, and then materialize them in memory
> in Arrow's format.
>
> Both formats are designed around the idiosyncrasies of the target medium:
> Parquet is not designed to support efficient random access because disks
> aren't good at that, but Arrow has fast random access as a core design
> principle, to give just one example.
>
> Henry
>
> > On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <
> sourav.mazumder00@gmail.com> wrote:
> >
> > Hi All,
> >
> > New to this. And still trying to figure out where exactly Arrow fits
> > in the ecosystem of various Big Data technologies.
> >
> > In that respect first thing which came to my mind is how does Arrow
> > compare with parquet.
> >
> > In my understanding Parquet also supports a very efficient columnar
> > format (with support for nested structure). It is already embraced
> > (supported) by various technologies like Impala (origin), Spark, Drill
> etc.
> >
> > The only think I see missing in Parquet is support for SIMD based
> > vectorized operations.
> >
> > Am I right or am I missing many other differences between Arrow and
> > parquet ?
> >
> > Regards,
> > Sourav
>
答复: Comparing with Parquet
Posted by "Chenliang (Liang, DataSight)" <ch...@huawei.com>.
In favor of Henry Robinson's points.
In addition. Arrow is suitable for exchanging data high efficiently, but the data size may just support TB level. Parquet can support more bigger data, but the performance couldn't support fast query.
So for PB level data and interactively query(second level), both couldn't solve?
Regards
Liang
-----邮件原件-----
发件人: Henry Robinson [mailto:henry@cloudera.com]
发送时间: 2016年2月26日 0:20
收件人: dev@arrow.apache.org
主题: Re: Comparing with Parquet
Think of Parquet as a format well-suited to writing very large datasets to disk, whereas Arrow is a format most suited to efficient storage in memory. You might read Parquet files from disk, and then materialize them in memory in Arrow's format.
Both formats are designed around the idiosyncrasies of the target medium: Parquet is not designed to support efficient random access because disks aren't good at that, but Arrow has fast random access as a core design principle, to give just one example.
Henry
> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <so...@gmail.com> wrote:
>
> Hi All,
>
> New to this. And still trying to figure out where exactly Arrow fits
> in the ecosystem of various Big Data technologies.
>
> In that respect first thing which came to my mind is how does Arrow
> compare with parquet.
>
> In my understanding Parquet also supports a very efficient columnar
> format (with support for nested structure). It is already embraced
> (supported) by various technologies like Impala (origin), Spark, Drill etc.
>
> The only think I see missing in Parquet is support for SIMD based
> vectorized operations.
>
> Am I right or am I missing many other differences between Arrow and
> parquet ?
>
> Regards,
> Sourav
Re: Comparing with Parquet
Posted by Jason Altekruse <al...@gmail.com>.
That being said, sometimes encodings can be complementary to processing.
Especially in the case of RLE, if a value is only stored once but stored in
a way that it represents a value shared across many rows, you only need to
do do the calculation once.
This type of optimization is something that I think would be useful to
include in the Arrow spec, especially in the case of sparse datasets.
Currently the java implementation of Arrow, which lacks support for
different encodings, will allocate memory to represent a possible value
(including four unused bytes for each null int value to get random access
to the non-null values) as well as a secondary vector to represent the
nullability of each value, even in the case where all values are null or
all values are non-null.
The approach has two downsides, obviously there is obvious redundancy that
is stored in full fidelity rather than being compressed, but it also
spreads out all of the information about what values are present throughout
the vector. We then end up having run a filter or a calculation on a bunch
of rows that should have just been a single evaluation if we had kept
better track of what we had written into the vector and appropriately
stored that information efficiently.
There is a trade-off to doing this kind of analysis on the fly mid-query,
as there is overhead to maintaining statistics about the batch, but as this
data is going to be read out of efficient disk formats like parquet in many
cases, it makes sense to preserve this information where it is useful for
processing.
On Thu, Feb 25, 2016 at 11:48 AM, Andrew Brust <
andrew.brust@bluebadgeinsights.com> wrote:
> Also extremely helpful; thank you!
>
> -----Original Message-----
> From: Reynold Xin [mailto:rxin@databricks.com]
> Sent: Thursday, February 25, 2016 2:46 PM
> To: dev@arrow.apache.org
> Subject: Re: Comparing with Parquet
>
> To put it even more layman, on-disk formats are typically designed for
> more permanent storage on disks/ssds, and as a result the format would want
> to reduce the size, because:
>
> 1. For some clusters, they are bottlenecked by the amount of disk space
> available. In these cases, you'd want to compress heavily the data.
>
> 2. Disks are slower than memory, and as a a result things might speed up
> if data is compressed (use more cpu cycles).
>
>
> For in-memory format, they are typically ephemeral, and have opposite
> characteristics.
>
>
>
>
> On Thu, Feb 25, 2016 at 11:27 AM, Andrew Brust <
> andrew.brust@bluebadgeinsights.com> wrote:
>
> > That's extremely helpful, thank you Todd.
> >
> > (And nice to "see" you again. I interviewed you years ago.)
> >
> > -----Original Message-----
> > From: Todd Lipcon [mailto:todd@cloudera.com]
> > Sent: Thursday, February 25, 2016 2:23 PM
> > To: dev@arrow.apache.org
> > Subject: Re: Comparing with Parquet
> >
> > I would say that another key difference is that Parquet puts a lot of
> > effort on encodings and compression, and Arrow is mostly about
> > efficient representation to directly run operators over. eg simple
> > arrays in memory vs bitpacked RLE-encoded data on disk.
> >
> > -Todd
> >
> > On Thu, Feb 25, 2016 at 11:20 AM, Andrew Brust <
> > andrew.brust@bluebadgeinsights.com> wrote:
> > > Is there a dumbed-down version of as summary for how and why in-mem
> > > and
> > on-disk formats differ? Is it mostly around aligning things for
> > SIMD/vectorization?
> > >
> > > There is probably some ignorance in my question, but I'm comfortable
> > > with that. :-)
> > >
> > > -----Original Message-----
> > > From: Wes McKinney [mailto:wes@cloudera.com]
> > > Sent: Thursday, February 25, 2016 12:12 PM
> > > To: dev@arrow.apache.org
> > > Subject: Re: Comparing with Parquet
> > >
> > > We wrote about this in a recent blog post:
> > >
> > > http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fas
> > > t- interoperable-in-memory-columnar-data-structure-standard/
> > >
> > > "Apache Parquet is a compact, efficient columnar data storage
> > > designed
> > for storing large amounts of data stored in HDFS. Arrow is an ideal
> > in-memory “container” for data that has been deserialized from a
> > Parquet file, and similarly in-memory Arrow data can be serialized to
> > Parquet and written out to a filesystem like HDFS or Amazon S3. Arrow
> > and Parquet are thus companion projects."
> > >
> > > For example, one of my personal motivations for being involved in
> > > both
> > Arrow and Parquet is to use Arrow as the in-memory container for data
> > deserialized from Parquet for use in Python and R.
> > >
> > > - Wes
> > >
> > > On Thu, Feb 25, 2016 at 8:20 AM, Henry Robinson <he...@cloudera.com>
> > wrote:
> > >> Think of Parquet as a format well-suited to writing very large
> > >> datasets
> > to disk, whereas Arrow is a format most suited to efficient storage in
> > memory. You might read Parquet files from disk, and then materialize
> > them in memory in Arrow's format.
> > >>
> > >> Both formats are designed around the idiosyncrasies of the target
> > medium: Parquet is not designed to support efficient random access
> > because disks aren't good at that, but Arrow has fast random access
> > as a core design principle, to give just one example.
> > >>
> > >> Henry
> > >>
> > >>> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <
> > sourav.mazumder00@gmail.com> wrote:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> New to this. And still trying to figure out where exactly Arrow
> > >>> fits in the ecosystem of various Big Data technologies.
> > >>>
> > >>> In that respect first thing which came to my mind is how does
> > >>> Arrow compare with parquet.
> > >>>
> > >>> In my understanding Parquet also supports a very efficient
> > >>> columnar format (with support for nested structure). It is already
> > >>> embraced
> > >>> (supported) by various technologies like Impala (origin), Spark,
> > >>> Drill
> > etc.
> > >>>
> > >>> The only think I see missing in Parquet is support for SIMD based
> > >>> vectorized operations.
> > >>>
> > >>> Am I right or am I missing many other differences between Arrow
> > >>> and parquet ?
> > >>>
> > >>> Regards,
> > >>> Sourav
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
Re: Comparing with Parquet
Posted by Sourav Mazumder <so...@gmail.com>.
Thanks to all you guys for the inputs/explanations.
Seems like it is pretty interesting project if it is complementing parquet
for in memory data storage structure for analytical processing as well as
random access within the same format. So far the formats which tried to
take care of random access (like HFile etc.) were not at all geared to
address analytical query requirements around aggregates/windowing etc.
Regards,
Sourav
On Thu, Feb 25, 2016 at 11:48 AM, Andrew Brust <
andrew.brust@bluebadgeinsights.com> wrote:
> Also extremely helpful; thank you!
>
> -----Original Message-----
> From: Reynold Xin [mailto:rxin@databricks.com]
> Sent: Thursday, February 25, 2016 2:46 PM
> To: dev@arrow.apache.org
> Subject: Re: Comparing with Parquet
>
> To put it even more layman, on-disk formats are typically designed for
> more permanent storage on disks/ssds, and as a result the format would want
> to reduce the size, because:
>
> 1. For some clusters, they are bottlenecked by the amount of disk space
> available. In these cases, you'd want to compress heavily the data.
>
> 2. Disks are slower than memory, and as a a result things might speed up
> if data is compressed (use more cpu cycles).
>
>
> For in-memory format, they are typically ephemeral, and have opposite
> characteristics.
>
>
>
>
> On Thu, Feb 25, 2016 at 11:27 AM, Andrew Brust <
> andrew.brust@bluebadgeinsights.com> wrote:
>
> > That's extremely helpful, thank you Todd.
> >
> > (And nice to "see" you again. I interviewed you years ago.)
> >
> > -----Original Message-----
> > From: Todd Lipcon [mailto:todd@cloudera.com]
> > Sent: Thursday, February 25, 2016 2:23 PM
> > To: dev@arrow.apache.org
> > Subject: Re: Comparing with Parquet
> >
> > I would say that another key difference is that Parquet puts a lot of
> > effort on encodings and compression, and Arrow is mostly about
> > efficient representation to directly run operators over. eg simple
> > arrays in memory vs bitpacked RLE-encoded data on disk.
> >
> > -Todd
> >
> > On Thu, Feb 25, 2016 at 11:20 AM, Andrew Brust <
> > andrew.brust@bluebadgeinsights.com> wrote:
> > > Is there a dumbed-down version of as summary for how and why in-mem
> > > and
> > on-disk formats differ? Is it mostly around aligning things for
> > SIMD/vectorization?
> > >
> > > There is probably some ignorance in my question, but I'm comfortable
> > > with that. :-)
> > >
> > > -----Original Message-----
> > > From: Wes McKinney [mailto:wes@cloudera.com]
> > > Sent: Thursday, February 25, 2016 12:12 PM
> > > To: dev@arrow.apache.org
> > > Subject: Re: Comparing with Parquet
> > >
> > > We wrote about this in a recent blog post:
> > >
> > > http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fas
> > > t- interoperable-in-memory-columnar-data-structure-standard/
> > >
> > > "Apache Parquet is a compact, efficient columnar data storage
> > > designed
> > for storing large amounts of data stored in HDFS. Arrow is an ideal
> > in-memory “container” for data that has been deserialized from a
> > Parquet file, and similarly in-memory Arrow data can be serialized to
> > Parquet and written out to a filesystem like HDFS or Amazon S3. Arrow
> > and Parquet are thus companion projects."
> > >
> > > For example, one of my personal motivations for being involved in
> > > both
> > Arrow and Parquet is to use Arrow as the in-memory container for data
> > deserialized from Parquet for use in Python and R.
> > >
> > > - Wes
> > >
> > > On Thu, Feb 25, 2016 at 8:20 AM, Henry Robinson <he...@cloudera.com>
> > wrote:
> > >> Think of Parquet as a format well-suited to writing very large
> > >> datasets
> > to disk, whereas Arrow is a format most suited to efficient storage in
> > memory. You might read Parquet files from disk, and then materialize
> > them in memory in Arrow's format.
> > >>
> > >> Both formats are designed around the idiosyncrasies of the target
> > medium: Parquet is not designed to support efficient random access
> > because disks aren't good at that, but Arrow has fast random access
> > as a core design principle, to give just one example.
> > >>
> > >> Henry
> > >>
> > >>> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <
> > sourav.mazumder00@gmail.com> wrote:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> New to this. And still trying to figure out where exactly Arrow
> > >>> fits in the ecosystem of various Big Data technologies.
> > >>>
> > >>> In that respect first thing which came to my mind is how does
> > >>> Arrow compare with parquet.
> > >>>
> > >>> In my understanding Parquet also supports a very efficient
> > >>> columnar format (with support for nested structure). It is already
> > >>> embraced
> > >>> (supported) by various technologies like Impala (origin), Spark,
> > >>> Drill
> > etc.
> > >>>
> > >>> The only think I see missing in Parquet is support for SIMD based
> > >>> vectorized operations.
> > >>>
> > >>> Am I right or am I missing many other differences between Arrow
> > >>> and parquet ?
> > >>>
> > >>> Regards,
> > >>> Sourav
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
RE: Comparing with Parquet
Posted by Andrew Brust <an...@bluebadgeinsights.com>.
Also extremely helpful; thank you!
-----Original Message-----
From: Reynold Xin [mailto:rxin@databricks.com]
Sent: Thursday, February 25, 2016 2:46 PM
To: dev@arrow.apache.org
Subject: Re: Comparing with Parquet
To put it even more layman, on-disk formats are typically designed for more permanent storage on disks/ssds, and as a result the format would want to reduce the size, because:
1. For some clusters, they are bottlenecked by the amount of disk space available. In these cases, you'd want to compress heavily the data.
2. Disks are slower than memory, and as a a result things might speed up if data is compressed (use more cpu cycles).
For in-memory format, they are typically ephemeral, and have opposite characteristics.
On Thu, Feb 25, 2016 at 11:27 AM, Andrew Brust < andrew.brust@bluebadgeinsights.com> wrote:
> That's extremely helpful, thank you Todd.
>
> (And nice to "see" you again. I interviewed you years ago.)
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Thursday, February 25, 2016 2:23 PM
> To: dev@arrow.apache.org
> Subject: Re: Comparing with Parquet
>
> I would say that another key difference is that Parquet puts a lot of
> effort on encodings and compression, and Arrow is mostly about
> efficient representation to directly run operators over. eg simple
> arrays in memory vs bitpacked RLE-encoded data on disk.
>
> -Todd
>
> On Thu, Feb 25, 2016 at 11:20 AM, Andrew Brust <
> andrew.brust@bluebadgeinsights.com> wrote:
> > Is there a dumbed-down version of as summary for how and why in-mem
> > and
> on-disk formats differ? Is it mostly around aligning things for
> SIMD/vectorization?
> >
> > There is probably some ignorance in my question, but I'm comfortable
> > with that. :-)
> >
> > -----Original Message-----
> > From: Wes McKinney [mailto:wes@cloudera.com]
> > Sent: Thursday, February 25, 2016 12:12 PM
> > To: dev@arrow.apache.org
> > Subject: Re: Comparing with Parquet
> >
> > We wrote about this in a recent blog post:
> >
> > http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fas
> > t- interoperable-in-memory-columnar-data-structure-standard/
> >
> > "Apache Parquet is a compact, efficient columnar data storage
> > designed
> for storing large amounts of data stored in HDFS. Arrow is an ideal
> in-memory “container” for data that has been deserialized from a
> Parquet file, and similarly in-memory Arrow data can be serialized to
> Parquet and written out to a filesystem like HDFS or Amazon S3. Arrow
> and Parquet are thus companion projects."
> >
> > For example, one of my personal motivations for being involved in
> > both
> Arrow and Parquet is to use Arrow as the in-memory container for data
> deserialized from Parquet for use in Python and R.
> >
> > - Wes
> >
> > On Thu, Feb 25, 2016 at 8:20 AM, Henry Robinson <he...@cloudera.com>
> wrote:
> >> Think of Parquet as a format well-suited to writing very large
> >> datasets
> to disk, whereas Arrow is a format most suited to efficient storage in
> memory. You might read Parquet files from disk, and then materialize
> them in memory in Arrow's format.
> >>
> >> Both formats are designed around the idiosyncrasies of the target
> medium: Parquet is not designed to support efficient random access
> because disks aren't good at that, but Arrow has fast random access
> as a core design principle, to give just one example.
> >>
> >> Henry
> >>
> >>> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <
> sourav.mazumder00@gmail.com> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> New to this. And still trying to figure out where exactly Arrow
> >>> fits in the ecosystem of various Big Data technologies.
> >>>
> >>> In that respect first thing which came to my mind is how does
> >>> Arrow compare with parquet.
> >>>
> >>> In my understanding Parquet also supports a very efficient
> >>> columnar format (with support for nested structure). It is already
> >>> embraced
> >>> (supported) by various technologies like Impala (origin), Spark,
> >>> Drill
> etc.
> >>>
> >>> The only think I see missing in Parquet is support for SIMD based
> >>> vectorized operations.
> >>>
> >>> Am I right or am I missing many other differences between Arrow
> >>> and parquet ?
> >>>
> >>> Regards,
> >>> Sourav
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Comparing with Parquet
Posted by Reynold Xin <rx...@databricks.com>.
To put it even more layman, on-disk formats are typically designed for more
permanent storage on disks/ssds, and as a result the format would want to
reduce the size, because:
1. For some clusters, they are bottlenecked by the amount of disk space
available. In these cases, you'd want to compress heavily the data.
2. Disks are slower than memory, and as a a result things might speed up if
data is compressed (use more cpu cycles).
For in-memory format, they are typically ephemeral, and have opposite
characteristics.
On Thu, Feb 25, 2016 at 11:27 AM, Andrew Brust <
andrew.brust@bluebadgeinsights.com> wrote:
> That's extremely helpful, thank you Todd.
>
> (And nice to "see" you again. I interviewed you years ago.)
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Thursday, February 25, 2016 2:23 PM
> To: dev@arrow.apache.org
> Subject: Re: Comparing with Parquet
>
> I would say that another key difference is that Parquet puts a lot of
> effort on encodings and compression, and Arrow is mostly about efficient
> representation to directly run operators over. eg simple arrays in memory
> vs bitpacked RLE-encoded data on disk.
>
> -Todd
>
> On Thu, Feb 25, 2016 at 11:20 AM, Andrew Brust <
> andrew.brust@bluebadgeinsights.com> wrote:
> > Is there a dumbed-down version of as summary for how and why in-mem and
> on-disk formats differ? Is it mostly around aligning things for
> SIMD/vectorization?
> >
> > There is probably some ignorance in my question, but I'm comfortable
> > with that. :-)
> >
> > -----Original Message-----
> > From: Wes McKinney [mailto:wes@cloudera.com]
> > Sent: Thursday, February 25, 2016 12:12 PM
> > To: dev@arrow.apache.org
> > Subject: Re: Comparing with Parquet
> >
> > We wrote about this in a recent blog post:
> >
> > http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fast-
> > interoperable-in-memory-columnar-data-structure-standard/
> >
> > "Apache Parquet is a compact, efficient columnar data storage designed
> for storing large amounts of data stored in HDFS. Arrow is an ideal
> in-memory “container” for data that has been deserialized from a Parquet
> file, and similarly in-memory Arrow data can be serialized to Parquet and
> written out to a filesystem like HDFS or Amazon S3. Arrow and Parquet are
> thus companion projects."
> >
> > For example, one of my personal motivations for being involved in both
> Arrow and Parquet is to use Arrow as the in-memory container for data
> deserialized from Parquet for use in Python and R.
> >
> > - Wes
> >
> > On Thu, Feb 25, 2016 at 8:20 AM, Henry Robinson <he...@cloudera.com>
> wrote:
> >> Think of Parquet as a format well-suited to writing very large datasets
> to disk, whereas Arrow is a format most suited to efficient storage in
> memory. You might read Parquet files from disk, and then materialize them
> in memory in Arrow's format.
> >>
> >> Both formats are designed around the idiosyncrasies of the target
> medium: Parquet is not designed to support efficient random access because
> disks aren't good at that, but Arrow has fast random access as a core
> design principle, to give just one example.
> >>
> >> Henry
> >>
> >>> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <
> sourav.mazumder00@gmail.com> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> New to this. And still trying to figure out where exactly Arrow fits
> >>> in the ecosystem of various Big Data technologies.
> >>>
> >>> In that respect first thing which came to my mind is how does Arrow
> >>> compare with parquet.
> >>>
> >>> In my understanding Parquet also supports a very efficient columnar
> >>> format (with support for nested structure). It is already embraced
> >>> (supported) by various technologies like Impala (origin), Spark, Drill
> etc.
> >>>
> >>> The only think I see missing in Parquet is support for SIMD based
> >>> vectorized operations.
> >>>
> >>> Am I right or am I missing many other differences between Arrow and
> >>> parquet ?
> >>>
> >>> Regards,
> >>> Sourav
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
RE: Comparing with Parquet
Posted by Andrew Brust <an...@bluebadgeinsights.com>.
That's extremely helpful, thank you Todd.
(And nice to "see" you again. I interviewed you years ago.)
-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Thursday, February 25, 2016 2:23 PM
To: dev@arrow.apache.org
Subject: Re: Comparing with Parquet
I would say that another key difference is that Parquet puts a lot of effort on encodings and compression, and Arrow is mostly about efficient representation to directly run operators over. eg simple arrays in memory vs bitpacked RLE-encoded data on disk.
-Todd
On Thu, Feb 25, 2016 at 11:20 AM, Andrew Brust <an...@bluebadgeinsights.com> wrote:
> Is there a dumbed-down version of as summary for how and why in-mem and on-disk formats differ? Is it mostly around aligning things for SIMD/vectorization?
>
> There is probably some ignorance in my question, but I'm comfortable
> with that. :-)
>
> -----Original Message-----
> From: Wes McKinney [mailto:wes@cloudera.com]
> Sent: Thursday, February 25, 2016 12:12 PM
> To: dev@arrow.apache.org
> Subject: Re: Comparing with Parquet
>
> We wrote about this in a recent blog post:
>
> http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fast-
> interoperable-in-memory-columnar-data-structure-standard/
>
> "Apache Parquet is a compact, efficient columnar data storage designed for storing large amounts of data stored in HDFS. Arrow is an ideal in-memory “container” for data that has been deserialized from a Parquet file, and similarly in-memory Arrow data can be serialized to Parquet and written out to a filesystem like HDFS or Amazon S3. Arrow and Parquet are thus companion projects."
>
> For example, one of my personal motivations for being involved in both Arrow and Parquet is to use Arrow as the in-memory container for data deserialized from Parquet for use in Python and R.
>
> - Wes
>
> On Thu, Feb 25, 2016 at 8:20 AM, Henry Robinson <he...@cloudera.com> wrote:
>> Think of Parquet as a format well-suited to writing very large datasets to disk, whereas Arrow is a format most suited to efficient storage in memory. You might read Parquet files from disk, and then materialize them in memory in Arrow's format.
>>
>> Both formats are designed around the idiosyncrasies of the target medium: Parquet is not designed to support efficient random access because disks aren't good at that, but Arrow has fast random access as a core design principle, to give just one example.
>>
>> Henry
>>
>>> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <so...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> New to this. And still trying to figure out where exactly Arrow fits
>>> in the ecosystem of various Big Data technologies.
>>>
>>> In that respect first thing which came to my mind is how does Arrow
>>> compare with parquet.
>>>
>>> In my understanding Parquet also supports a very efficient columnar
>>> format (with support for nested structure). It is already embraced
>>> (supported) by various technologies like Impala (origin), Spark, Drill etc.
>>>
>>> The only think I see missing in Parquet is support for SIMD based
>>> vectorized operations.
>>>
>>> Am I right or am I missing many other differences between Arrow and
>>> parquet ?
>>>
>>> Regards,
>>> Sourav
--
Todd Lipcon
Software Engineer, Cloudera
Re: Comparing with Parquet
Posted by Todd Lipcon <to...@cloudera.com>.
I would say that another key difference is that Parquet puts a lot of
effort on encodings and compression, and Arrow is mostly about
efficient representation to directly run operators over. eg simple
arrays in memory vs bitpacked RLE-encoded data on disk.
-Todd
On Thu, Feb 25, 2016 at 11:20 AM, Andrew Brust
<an...@bluebadgeinsights.com> wrote:
> Is there a dumbed-down version of as summary for how and why in-mem and on-disk formats differ? Is it mostly around aligning things for SIMD/vectorization?
>
> There is probably some ignorance in my question, but I'm comfortable with that. :-)
>
> -----Original Message-----
> From: Wes McKinney [mailto:wes@cloudera.com]
> Sent: Thursday, February 25, 2016 12:12 PM
> To: dev@arrow.apache.org
> Subject: Re: Comparing with Parquet
>
> We wrote about this in a recent blog post:
>
> http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fast-interoperable-in-memory-columnar-data-structure-standard/
>
> "Apache Parquet is a compact, efficient columnar data storage designed for storing large amounts of data stored in HDFS. Arrow is an ideal in-memory “container” for data that has been deserialized from a Parquet file, and similarly in-memory Arrow data can be serialized to Parquet and written out to a filesystem like HDFS or Amazon S3. Arrow and Parquet are thus companion projects."
>
> For example, one of my personal motivations for being involved in both Arrow and Parquet is to use Arrow as the in-memory container for data deserialized from Parquet for use in Python and R.
>
> - Wes
>
> On Thu, Feb 25, 2016 at 8:20 AM, Henry Robinson <he...@cloudera.com> wrote:
>> Think of Parquet as a format well-suited to writing very large datasets to disk, whereas Arrow is a format most suited to efficient storage in memory. You might read Parquet files from disk, and then materialize them in memory in Arrow's format.
>>
>> Both formats are designed around the idiosyncrasies of the target medium: Parquet is not designed to support efficient random access because disks aren't good at that, but Arrow has fast random access as a core design principle, to give just one example.
>>
>> Henry
>>
>>> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <so...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> New to this. And still trying to figure out where exactly Arrow fits
>>> in the ecosystem of various Big Data technologies.
>>>
>>> In that respect first thing which came to my mind is how does Arrow
>>> compare with parquet.
>>>
>>> In my understanding Parquet also supports a very efficient columnar
>>> format (with support for nested structure). It is already embraced
>>> (supported) by various technologies like Impala (origin), Spark, Drill etc.
>>>
>>> The only think I see missing in Parquet is support for SIMD based
>>> vectorized operations.
>>>
>>> Am I right or am I missing many other differences between Arrow and
>>> parquet ?
>>>
>>> Regards,
>>> Sourav
--
Todd Lipcon
Software Engineer, Cloudera
RE: Comparing with Parquet
Posted by Andrew Brust <an...@bluebadgeinsights.com>.
Is there a dumbed-down version of as summary for how and why in-mem and on-disk formats differ? Is it mostly around aligning things for SIMD/vectorization?
There is probably some ignorance in my question, but I'm comfortable with that. :-)
-----Original Message-----
From: Wes McKinney [mailto:wes@cloudera.com]
Sent: Thursday, February 25, 2016 12:12 PM
To: dev@arrow.apache.org
Subject: Re: Comparing with Parquet
We wrote about this in a recent blog post:
http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fast-interoperable-in-memory-columnar-data-structure-standard/
"Apache Parquet is a compact, efficient columnar data storage designed for storing large amounts of data stored in HDFS. Arrow is an ideal in-memory “container” for data that has been deserialized from a Parquet file, and similarly in-memory Arrow data can be serialized to Parquet and written out to a filesystem like HDFS or Amazon S3. Arrow and Parquet are thus companion projects."
For example, one of my personal motivations for being involved in both Arrow and Parquet is to use Arrow as the in-memory container for data deserialized from Parquet for use in Python and R.
- Wes
On Thu, Feb 25, 2016 at 8:20 AM, Henry Robinson <he...@cloudera.com> wrote:
> Think of Parquet as a format well-suited to writing very large datasets to disk, whereas Arrow is a format most suited to efficient storage in memory. You might read Parquet files from disk, and then materialize them in memory in Arrow's format.
>
> Both formats are designed around the idiosyncrasies of the target medium: Parquet is not designed to support efficient random access because disks aren't good at that, but Arrow has fast random access as a core design principle, to give just one example.
>
> Henry
>
>> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <so...@gmail.com> wrote:
>>
>> Hi All,
>>
>> New to this. And still trying to figure out where exactly Arrow fits
>> in the ecosystem of various Big Data technologies.
>>
>> In that respect first thing which came to my mind is how does Arrow
>> compare with parquet.
>>
>> In my understanding Parquet also supports a very efficient columnar
>> format (with support for nested structure). It is already embraced
>> (supported) by various technologies like Impala (origin), Spark, Drill etc.
>>
>> The only think I see missing in Parquet is support for SIMD based
>> vectorized operations.
>>
>> Am I right or am I missing many other differences between Arrow and
>> parquet ?
>>
>> Regards,
>> Sourav
Re: Comparing with Parquet
Posted by Wes McKinney <we...@cloudera.com>.
We wrote about this in a recent blog post:
http://blog.cloudera.com/blog/2016/02/introducing-apache-arrow-a-fast-interoperable-in-memory-columnar-data-structure-standard/
"Apache Parquet is a compact, efficient columnar data storage designed
for storing large amounts of data stored in HDFS. Arrow is an ideal
in-memory “container” for data that has been deserialized from a
Parquet file, and similarly in-memory Arrow data can be serialized to
Parquet and written out to a filesystem like HDFS or Amazon S3. Arrow
and Parquet are thus companion projects."
For example, one of my personal motivations for being involved in both
Arrow and Parquet is to use Arrow as the in-memory container for data
deserialized from Parquet for use in Python and R.
- Wes
On Thu, Feb 25, 2016 at 8:20 AM, Henry Robinson <he...@cloudera.com> wrote:
> Think of Parquet as a format well-suited to writing very large datasets to disk, whereas Arrow is a format most suited to efficient storage in memory. You might read Parquet files from disk, and then materialize them in memory in Arrow's format.
>
> Both formats are designed around the idiosyncrasies of the target medium: Parquet is not designed to support efficient random access because disks aren't good at that, but Arrow has fast random access as a core design principle, to give just one example.
>
> Henry
>
>> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <so...@gmail.com> wrote:
>>
>> Hi All,
>>
>> New to this. And still trying to figure out where exactly Arrow fits in the
>> ecosystem of various Big Data technologies.
>>
>> In that respect first thing which came to my mind is how does Arrow compare
>> with parquet.
>>
>> In my understanding Parquet also supports a very efficient columnar format
>> (with support for nested structure). It is already embraced (supported) by
>> various technologies like Impala (origin), Spark, Drill etc.
>>
>> The only think I see missing in Parquet is support for SIMD based
>> vectorized operations.
>>
>> Am I right or am I missing many other differences between Arrow and parquet
>> ?
>>
>> Regards,
>> Sourav
Re: Comparing with Parquet
Posted by Henry Robinson <he...@cloudera.com>.
Think of Parquet as a format well-suited to writing very large datasets to disk, whereas Arrow is a format most suited to efficient storage in memory. You might read Parquet files from disk, and then materialize them in memory in Arrow's format.
Both formats are designed around the idiosyncrasies of the target medium: Parquet is not designed to support efficient random access because disks aren't good at that, but Arrow has fast random access as a core design principle, to give just one example.
Henry
> On Feb 25, 2016, at 8:10 AM, Sourav Mazumder <so...@gmail.com> wrote:
>
> Hi All,
>
> New to this. And still trying to figure out where exactly Arrow fits in the
> ecosystem of various Big Data technologies.
>
> In that respect first thing which came to my mind is how does Arrow compare
> with parquet.
>
> In my understanding Parquet also supports a very efficient columnar format
> (with support for nested structure). It is already embraced (supported) by
> various technologies like Impala (origin), Spark, Drill etc.
>
> The only think I see missing in Parquet is support for SIMD based
> vectorized operations.
>
> Am I right or am I missing many other differences between Arrow and parquet
> ?
>
> Regards,
> Sourav