You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Tim Swast <sw...@google.com.INVALID> on 2019/06/11 21:53:00 UTC
Avro to Arrow?
Hi Arrow and Avro devs,
I've been investigating some performance issues with the BigQuery Storage
API (https://github.com/googleapis/google-cloud-python/issues/7805), and
have identified that the vast majority of time is spent decoding Avro into
pandas dataframes.
<https://github.com/googleapis/google-cloud-python/issues/7805>
I've done some initial experiments by hand written parsers (inspired by
https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
dramatic improvement in time spent parsing.
I'm considering releasing this as a separate package for the following
reasons:
- Code generation + Numba is a bit of an unproven technique for parsers,
so I'd like to treat this as an experiment rather than "the" package to use
to parse Avro from Python.
- I don't need to handle the full Avro spec for this experiment.
Importantly, BQ Storage API only uses a schemaless reader (since the schema
is output only once, and omitted for subsequent protobuf messages) and
doesn't use any compression.
That said, I'm open to contributing this to either pyarrow or avro if
there's interest.
If the answer is "no" (as I suspect it is) and I don't contribute it now,
the package will be clearly identified as a fork of the Apache Avro project
and licensed Apache 2.0, so it should be easy to pull in once the
techniques are proven.
* • **Tim Swast*
* • *Software Friendliness Engineer
* • *Google Cloud Developer Relations
* • *Seattle, WA, USA
Re: Avro to Arrow?
Posted by Wes McKinney <we...@gmail.com>.
hi Tim,
I should think that the reader API should support deserializing a blob
of schemaless Avro records as an Arrow record batch, or even feeding
the reader one serialized record at a time to build a record batch
incrementally
- Wes
On Wed, Jun 12, 2019 at 1:25 PM Tim Swast <sw...@google.com.invalid> wrote:
>
> > Let me know if you want to collaborate on it.
>
> Thanks Micah.
>
> What are your thoughts on reading schemaless Avro bytes? One of the reasons
> I have started experimenting with the fork is that fastavro had trouble
> reading more than one row at a time from a schemaless reader.
>
> * • **Tim Swast*
> * • *Software Friendliness Engineer
> * • *Google Cloud Developer Relations
> * • *Seattle, WA, USA
>
>
> On Tue, Jun 11, 2019 at 10:29 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Hi Tim,
> > The avro support in C++ has been on my backlog for a while. I'm going to
> > try to take the first few steps towards this over the next couple of days.
> > Let me know if you want to collaborate on it. C++ is a lot nicer now then
> > it was 8 years ago :)
> >
> > Cheers,
> > Micah
> >
> > On Tue, Jun 11, 2019 at 6:40 PM Tim Swast <sw...@google.com.invalid>
> > wrote:
> >
> > > Thanks for the advice, Wes.
> > >
> > > Unfortunately, I am about 8 years out of practice for writing any C++
> > > (which was part of the appeal of numba to me). Sounds like I should
> > refresh
> > > my skills. I like the idea of write one, have good performance
> > everywhere.
> > >
> > > On Tue, Jun 11, 2019 at 3:40 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > >
> > > > Hi Tim,
> > > >
> > > > I'd ideally like to see the work done in the Arrow C++ library so that
> > it
> > > > can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB).
> > > This
> > > > also means a larger labor pool of individuals to help improve and
> > > maintain
> > > > the software. There was a stalled PR around this a time back (check out
> > > the
> > > > Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
> > > > might be more expedient to fork parts of Apache Avro and do all the
> > > > development inside a single codebase.
> > > >
> > > > There's a lot of folks that can provide feedback should you choose to
> > go
> > > > down this route.
> > > >
> > > > Thanks
> > > > Wes
> > > >
> > > > On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid>
> > > wrote:
> > > >
> > > > > Hi Arrow and Avro devs,
> > > > >
> > > > > I've been investigating some performance issues with the BigQuery
> > > Storage
> > > > > API (https://github.com/googleapis/google-cloud-python/issues/7805),
> > > and
> > > > > have identified that the vast majority of time is spent decoding Avro
> > > > into
> > > > > pandas dataframes.
> > > > > <https://github.com/googleapis/google-cloud-python/issues/7805>
> > > > > I've done some initial experiments by hand written parsers (inspired
> > by
> > > > > https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> > > > > dramatic improvement in time spent parsing.
> > > > >
> > > > > I'm considering releasing this as a separate package for the
> > following
> > > > > reasons:
> > > > >
> > > > > - Code generation + Numba is a bit of an unproven technique for
> > > > parsers,
> > > > > so I'd like to treat this as an experiment rather than "the"
> > package
> > > > to
> > > > > use
> > > > > to parse Avro from Python.
> > > > > - I don't need to handle the full Avro spec for this experiment.
> > > > > Importantly, BQ Storage API only uses a schemaless reader (since
> > the
> > > > > schema
> > > > > is output only once, and omitted for subsequent protobuf messages)
> > > and
> > > > > doesn't use any compression.
> > > > >
> > > > > That said, I'm open to contributing this to either pyarrow or avro if
> > > > > there's interest.
> > > > >
> > > > > If the answer is "no" (as I suspect it is) and I don't contribute it
> > > now,
> > > > > the package will be clearly identified as a fork of the Apache Avro
> > > > project
> > > > > and licensed Apache 2.0, so it should be easy to pull in once the
> > > > > techniques are proven.
> > > > >
> > > > > * • **Tim Swast*
> > > > > * • *Software Friendliness Engineer
> > > > > * • *Google Cloud Developer Relations
> > > > > * • *Seattle, WA, USA
> > > > >
> > > >
> > > --
> > > * • **Tim Swast*
> > > * • *Software Friendliness Engineer
> > > * • *Google Cloud Developer Relations
> > > * • *Seattle, WA, USA
> > >
> >
Re: Avro to Arrow?
Posted by Wes McKinney <we...@gmail.com>.
hi Tim,
I should think that the reader API should support deserializing a blob
of schemaless Avro records as an Arrow record batch, or even feeding
the reader one serialized record at a time to build a record batch
incrementally
- Wes
On Wed, Jun 12, 2019 at 1:25 PM Tim Swast <sw...@google.com.invalid> wrote:
>
> > Let me know if you want to collaborate on it.
>
> Thanks Micah.
>
> What are your thoughts on reading schemaless Avro bytes? One of the reasons
> I have started experimenting with the fork is that fastavro had trouble
> reading more than one row at a time from a schemaless reader.
>
> * • **Tim Swast*
> * • *Software Friendliness Engineer
> * • *Google Cloud Developer Relations
> * • *Seattle, WA, USA
>
>
> On Tue, Jun 11, 2019 at 10:29 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Hi Tim,
> > The avro support in C++ has been on my backlog for a while. I'm going to
> > try to take the first few steps towards this over the next couple of days.
> > Let me know if you want to collaborate on it. C++ is a lot nicer now then
> > it was 8 years ago :)
> >
> > Cheers,
> > Micah
> >
> > On Tue, Jun 11, 2019 at 6:40 PM Tim Swast <sw...@google.com.invalid>
> > wrote:
> >
> > > Thanks for the advice, Wes.
> > >
> > > Unfortunately, I am about 8 years out of practice for writing any C++
> > > (which was part of the appeal of numba to me). Sounds like I should
> > refresh
> > > my skills. I like the idea of write one, have good performance
> > everywhere.
> > >
> > > On Tue, Jun 11, 2019 at 3:40 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > >
> > > > Hi Tim,
> > > >
> > > > I'd ideally like to see the work done in the Arrow C++ library so that
> > it
> > > > can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB).
> > > This
> > > > also means a larger labor pool of individuals to help improve and
> > > maintain
> > > > the software. There was a stalled PR around this a time back (check out
> > > the
> > > > Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
> > > > might be more expedient to fork parts of Apache Avro and do all the
> > > > development inside a single codebase.
> > > >
> > > > There's a lot of folks that can provide feedback should you choose to
> > go
> > > > down this route.
> > > >
> > > > Thanks
> > > > Wes
> > > >
> > > > On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid>
> > > wrote:
> > > >
> > > > > Hi Arrow and Avro devs,
> > > > >
> > > > > I've been investigating some performance issues with the BigQuery
> > > Storage
> > > > > API (https://github.com/googleapis/google-cloud-python/issues/7805),
> > > and
> > > > > have identified that the vast majority of time is spent decoding Avro
> > > > into
> > > > > pandas dataframes.
> > > > > <https://github.com/googleapis/google-cloud-python/issues/7805>
> > > > > I've done some initial experiments by hand written parsers (inspired
> > by
> > > > > https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> > > > > dramatic improvement in time spent parsing.
> > > > >
> > > > > I'm considering releasing this as a separate package for the
> > following
> > > > > reasons:
> > > > >
> > > > > - Code generation + Numba is a bit of an unproven technique for
> > > > parsers,
> > > > > so I'd like to treat this as an experiment rather than "the"
> > package
> > > > to
> > > > > use
> > > > > to parse Avro from Python.
> > > > > - I don't need to handle the full Avro spec for this experiment.
> > > > > Importantly, BQ Storage API only uses a schemaless reader (since
> > the
> > > > > schema
> > > > > is output only once, and omitted for subsequent protobuf messages)
> > > and
> > > > > doesn't use any compression.
> > > > >
> > > > > That said, I'm open to contributing this to either pyarrow or avro if
> > > > > there's interest.
> > > > >
> > > > > If the answer is "no" (as I suspect it is) and I don't contribute it
> > > now,
> > > > > the package will be clearly identified as a fork of the Apache Avro
> > > > project
> > > > > and licensed Apache 2.0, so it should be easy to pull in once the
> > > > > techniques are proven.
> > > > >
> > > > > * • **Tim Swast*
> > > > > * • *Software Friendliness Engineer
> > > > > * • *Google Cloud Developer Relations
> > > > > * • *Seattle, WA, USA
> > > > >
> > > >
> > > --
> > > * • **Tim Swast*
> > > * • *Software Friendliness Engineer
> > > * • *Google Cloud Developer Relations
> > > * • *Seattle, WA, USA
> > >
> >
Re: Avro to Arrow?
Posted by Tim Swast <sw...@google.com.INVALID>.
> Let me know if you want to collaborate on it.
Thanks Micah.
What are your thoughts on reading schemaless Avro bytes? One of the reasons
I have started experimenting with the fork is that fastavro had trouble
reading more than one row at a time from a schemaless reader.
* • **Tim Swast*
* • *Software Friendliness Engineer
* • *Google Cloud Developer Relations
* • *Seattle, WA, USA
On Tue, Jun 11, 2019 at 10:29 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Tim,
> The avro support in C++ has been on my backlog for a while. I'm going to
> try to take the first few steps towards this over the next couple of days.
> Let me know if you want to collaborate on it. C++ is a lot nicer now then
> it was 8 years ago :)
>
> Cheers,
> Micah
>
> On Tue, Jun 11, 2019 at 6:40 PM Tim Swast <sw...@google.com.invalid>
> wrote:
>
> > Thanks for the advice, Wes.
> >
> > Unfortunately, I am about 8 years out of practice for writing any C++
> > (which was part of the appeal of numba to me). Sounds like I should
> refresh
> > my skills. I like the idea of write one, have good performance
> everywhere.
> >
> > On Tue, Jun 11, 2019 at 3:40 PM Wes McKinney <we...@gmail.com>
> wrote:
> >
> > > Hi Tim,
> > >
> > > I'd ideally like to see the work done in the Arrow C++ library so that
> it
> > > can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB).
> > This
> > > also means a larger labor pool of individuals to help improve and
> > maintain
> > > the software. There was a stalled PR around this a time back (check out
> > the
> > > Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
> > > might be more expedient to fork parts of Apache Avro and do all the
> > > development inside a single codebase.
> > >
> > > There's a lot of folks that can provide feedback should you choose to
> go
> > > down this route.
> > >
> > > Thanks
> > > Wes
> > >
> > > On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid>
> > wrote:
> > >
> > > > Hi Arrow and Avro devs,
> > > >
> > > > I've been investigating some performance issues with the BigQuery
> > Storage
> > > > API (https://github.com/googleapis/google-cloud-python/issues/7805),
> > and
> > > > have identified that the vast majority of time is spent decoding Avro
> > > into
> > > > pandas dataframes.
> > > > <https://github.com/googleapis/google-cloud-python/issues/7805>
> > > > I've done some initial experiments by hand written parsers (inspired
> by
> > > > https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> > > > dramatic improvement in time spent parsing.
> > > >
> > > > I'm considering releasing this as a separate package for the
> following
> > > > reasons:
> > > >
> > > > - Code generation + Numba is a bit of an unproven technique for
> > > parsers,
> > > > so I'd like to treat this as an experiment rather than "the"
> package
> > > to
> > > > use
> > > > to parse Avro from Python.
> > > > - I don't need to handle the full Avro spec for this experiment.
> > > > Importantly, BQ Storage API only uses a schemaless reader (since
> the
> > > > schema
> > > > is output only once, and omitted for subsequent protobuf messages)
> > and
> > > > doesn't use any compression.
> > > >
> > > > That said, I'm open to contributing this to either pyarrow or avro if
> > > > there's interest.
> > > >
> > > > If the answer is "no" (as I suspect it is) and I don't contribute it
> > now,
> > > > the package will be clearly identified as a fork of the Apache Avro
> > > project
> > > > and licensed Apache 2.0, so it should be easy to pull in once the
> > > > techniques are proven.
> > > >
> > > > * • **Tim Swast*
> > > > * • *Software Friendliness Engineer
> > > > * • *Google Cloud Developer Relations
> > > > * • *Seattle, WA, USA
> > > >
> > >
> > --
> > * • **Tim Swast*
> > * • *Software Friendliness Engineer
> > * • *Google Cloud Developer Relations
> > * • *Seattle, WA, USA
> >
>
Re: Avro to Arrow?
Posted by Tim Swast <sw...@google.com.INVALID>.
> Let me know if you want to collaborate on it.
Thanks Micah.
What are your thoughts on reading schemaless Avro bytes? One of the reasons
I have started experimenting with the fork is that fastavro had trouble
reading more than one row at a time from a schemaless reader.
* • **Tim Swast*
* • *Software Friendliness Engineer
* • *Google Cloud Developer Relations
* • *Seattle, WA, USA
On Tue, Jun 11, 2019 at 10:29 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Tim,
> The avro support in C++ has been on my backlog for a while. I'm going to
> try to take the first few steps towards this over the next couple of days.
> Let me know if you want to collaborate on it. C++ is a lot nicer now then
> it was 8 years ago :)
>
> Cheers,
> Micah
>
> On Tue, Jun 11, 2019 at 6:40 PM Tim Swast <sw...@google.com.invalid>
> wrote:
>
> > Thanks for the advice, Wes.
> >
> > Unfortunately, I am about 8 years out of practice for writing any C++
> > (which was part of the appeal of numba to me). Sounds like I should
> refresh
> > my skills. I like the idea of write one, have good performance
> everywhere.
> >
> > On Tue, Jun 11, 2019 at 3:40 PM Wes McKinney <we...@gmail.com>
> wrote:
> >
> > > Hi Tim,
> > >
> > > I'd ideally like to see the work done in the Arrow C++ library so that
> it
> > > can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB).
> > This
> > > also means a larger labor pool of individuals to help improve and
> > maintain
> > > the software. There was a stalled PR around this a time back (check out
> > the
> > > Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
> > > might be more expedient to fork parts of Apache Avro and do all the
> > > development inside a single codebase.
> > >
> > > There's a lot of folks that can provide feedback should you choose to
> go
> > > down this route.
> > >
> > > Thanks
> > > Wes
> > >
> > > On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid>
> > wrote:
> > >
> > > > Hi Arrow and Avro devs,
> > > >
> > > > I've been investigating some performance issues with the BigQuery
> > Storage
> > > > API (https://github.com/googleapis/google-cloud-python/issues/7805),
> > and
> > > > have identified that the vast majority of time is spent decoding Avro
> > > into
> > > > pandas dataframes.
> > > > <https://github.com/googleapis/google-cloud-python/issues/7805>
> > > > I've done some initial experiments by hand written parsers (inspired
> by
> > > > https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> > > > dramatic improvement in time spent parsing.
> > > >
> > > > I'm considering releasing this as a separate package for the
> following
> > > > reasons:
> > > >
> > > > - Code generation + Numba is a bit of an unproven technique for
> > > parsers,
> > > > so I'd like to treat this as an experiment rather than "the"
> package
> > > to
> > > > use
> > > > to parse Avro from Python.
> > > > - I don't need to handle the full Avro spec for this experiment.
> > > > Importantly, BQ Storage API only uses a schemaless reader (since
> the
> > > > schema
> > > > is output only once, and omitted for subsequent protobuf messages)
> > and
> > > > doesn't use any compression.
> > > >
> > > > That said, I'm open to contributing this to either pyarrow or avro if
> > > > there's interest.
> > > >
> > > > If the answer is "no" (as I suspect it is) and I don't contribute it
> > now,
> > > > the package will be clearly identified as a fork of the Apache Avro
> > > project
> > > > and licensed Apache 2.0, so it should be easy to pull in once the
> > > > techniques are proven.
> > > >
> > > > * • **Tim Swast*
> > > > * • *Software Friendliness Engineer
> > > > * • *Google Cloud Developer Relations
> > > > * • *Seattle, WA, USA
> > > >
> > >
> > --
> > * • **Tim Swast*
> > * • *Software Friendliness Engineer
> > * • *Google Cloud Developer Relations
> > * • *Seattle, WA, USA
> >
>
Re: Avro to Arrow?
Posted by Micah Kornfield <em...@gmail.com>.
Hi Tim,
The avro support in C++ has been on my backlog for a while. I'm going to
try to take the first few steps towards this over the next couple of days.
Let me know if you want to collaborate on it. C++ is a lot nicer now then
it was 8 years ago :)
Cheers,
Micah
On Tue, Jun 11, 2019 at 6:40 PM Tim Swast <sw...@google.com.invalid> wrote:
> Thanks for the advice, Wes.
>
> Unfortunately, I am about 8 years out of practice for writing any C++
> (which was part of the appeal of numba to me). Sounds like I should refresh
> my skills. I like the idea of write one, have good performance everywhere.
>
> On Tue, Jun 11, 2019 at 3:40 PM Wes McKinney <we...@gmail.com> wrote:
>
> > Hi Tim,
> >
> > I'd ideally like to see the work done in the Arrow C++ library so that it
> > can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB).
> This
> > also means a larger labor pool of individuals to help improve and
> maintain
> > the software. There was a stalled PR around this a time back (check out
> the
> > Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
> > might be more expedient to fork parts of Apache Avro and do all the
> > development inside a single codebase.
> >
> > There's a lot of folks that can provide feedback should you choose to go
> > down this route.
> >
> > Thanks
> > Wes
> >
> > On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid>
> wrote:
> >
> > > Hi Arrow and Avro devs,
> > >
> > > I've been investigating some performance issues with the BigQuery
> Storage
> > > API (https://github.com/googleapis/google-cloud-python/issues/7805),
> and
> > > have identified that the vast majority of time is spent decoding Avro
> > into
> > > pandas dataframes.
> > > <https://github.com/googleapis/google-cloud-python/issues/7805>
> > > I've done some initial experiments by hand written parsers (inspired by
> > > https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> > > dramatic improvement in time spent parsing.
> > >
> > > I'm considering releasing this as a separate package for the following
> > > reasons:
> > >
> > > - Code generation + Numba is a bit of an unproven technique for
> > parsers,
> > > so I'd like to treat this as an experiment rather than "the" package
> > to
> > > use
> > > to parse Avro from Python.
> > > - I don't need to handle the full Avro spec for this experiment.
> > > Importantly, BQ Storage API only uses a schemaless reader (since the
> > > schema
> > > is output only once, and omitted for subsequent protobuf messages)
> and
> > > doesn't use any compression.
> > >
> > > That said, I'm open to contributing this to either pyarrow or avro if
> > > there's interest.
> > >
> > > If the answer is "no" (as I suspect it is) and I don't contribute it
> now,
> > > the package will be clearly identified as a fork of the Apache Avro
> > project
> > > and licensed Apache 2.0, so it should be easy to pull in once the
> > > techniques are proven.
> > >
> > > * • **Tim Swast*
> > > * • *Software Friendliness Engineer
> > > * • *Google Cloud Developer Relations
> > > * • *Seattle, WA, USA
> > >
> >
> --
> * • **Tim Swast*
> * • *Software Friendliness Engineer
> * • *Google Cloud Developer Relations
> * • *Seattle, WA, USA
>
Re: Avro to Arrow?
Posted by Micah Kornfield <em...@gmail.com>.
Hi Tim,
The avro support in C++ has been on my backlog for a while. I'm going to
try to take the first few steps towards this over the next couple of days.
Let me know if you want to collaborate on it. C++ is a lot nicer now then
it was 8 years ago :)
Cheers,
Micah
On Tue, Jun 11, 2019 at 6:40 PM Tim Swast <sw...@google.com.invalid> wrote:
> Thanks for the advice, Wes.
>
> Unfortunately, I am about 8 years out of practice for writing any C++
> (which was part of the appeal of numba to me). Sounds like I should refresh
> my skills. I like the idea of write one, have good performance everywhere.
>
> On Tue, Jun 11, 2019 at 3:40 PM Wes McKinney <we...@gmail.com> wrote:
>
> > Hi Tim,
> >
> > I'd ideally like to see the work done in the Arrow C++ library so that it
> > can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB).
> This
> > also means a larger labor pool of individuals to help improve and
> maintain
> > the software. There was a stalled PR around this a time back (check out
> the
> > Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
> > might be more expedient to fork parts of Apache Avro and do all the
> > development inside a single codebase.
> >
> > There's a lot of folks that can provide feedback should you choose to go
> > down this route.
> >
> > Thanks
> > Wes
> >
> > On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid>
> wrote:
> >
> > > Hi Arrow and Avro devs,
> > >
> > > I've been investigating some performance issues with the BigQuery
> Storage
> > > API (https://github.com/googleapis/google-cloud-python/issues/7805),
> and
> > > have identified that the vast majority of time is spent decoding Avro
> > into
> > > pandas dataframes.
> > > <https://github.com/googleapis/google-cloud-python/issues/7805>
> > > I've done some initial experiments by hand written parsers (inspired by
> > > https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> > > dramatic improvement in time spent parsing.
> > >
> > > I'm considering releasing this as a separate package for the following
> > > reasons:
> > >
> > > - Code generation + Numba is a bit of an unproven technique for
> > parsers,
> > > so I'd like to treat this as an experiment rather than "the" package
> > to
> > > use
> > > to parse Avro from Python.
> > > - I don't need to handle the full Avro spec for this experiment.
> > > Importantly, BQ Storage API only uses a schemaless reader (since the
> > > schema
> > > is output only once, and omitted for subsequent protobuf messages)
> and
> > > doesn't use any compression.
> > >
> > > That said, I'm open to contributing this to either pyarrow or avro if
> > > there's interest.
> > >
> > > If the answer is "no" (as I suspect it is) and I don't contribute it
> now,
> > > the package will be clearly identified as a fork of the Apache Avro
> > project
> > > and licensed Apache 2.0, so it should be easy to pull in once the
> > > techniques are proven.
> > >
> > > * • **Tim Swast*
> > > * • *Software Friendliness Engineer
> > > * • *Google Cloud Developer Relations
> > > * • *Seattle, WA, USA
> > >
> >
> --
> * • **Tim Swast*
> * • *Software Friendliness Engineer
> * • *Google Cloud Developer Relations
> * • *Seattle, WA, USA
>
Re: Avro to Arrow?
Posted by Tim Swast <sw...@google.com.INVALID>.
Thanks for the advice, Wes.
Unfortunately, I am about 8 years out of practice for writing any C++
(which was part of the appeal of numba to me). Sounds like I should refresh
my skills. I like the idea of write one, have good performance everywhere.
On Tue, Jun 11, 2019 at 3:40 PM Wes McKinney <we...@gmail.com> wrote:
> Hi Tim,
>
> I'd ideally like to see the work done in the Arrow C++ library so that it
> can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB). This
> also means a larger labor pool of individuals to help improve and maintain
> the software. There was a stalled PR around this a time back (check out the
> Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
> might be more expedient to fork parts of Apache Avro and do all the
> development inside a single codebase.
>
> There's a lot of folks that can provide feedback should you choose to go
> down this route.
>
> Thanks
> Wes
>
> On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid> wrote:
>
> > Hi Arrow and Avro devs,
> >
> > I've been investigating some performance issues with the BigQuery Storage
> > API (https://github.com/googleapis/google-cloud-python/issues/7805), and
> > have identified that the vast majority of time is spent decoding Avro
> into
> > pandas dataframes.
> > <https://github.com/googleapis/google-cloud-python/issues/7805>
> > I've done some initial experiments by hand written parsers (inspired by
> > https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> > dramatic improvement in time spent parsing.
> >
> > I'm considering releasing this as a separate package for the following
> > reasons:
> >
> > - Code generation + Numba is a bit of an unproven technique for
> parsers,
> > so I'd like to treat this as an experiment rather than "the" package
> to
> > use
> > to parse Avro from Python.
> > - I don't need to handle the full Avro spec for this experiment.
> > Importantly, BQ Storage API only uses a schemaless reader (since the
> > schema
> > is output only once, and omitted for subsequent protobuf messages) and
> > doesn't use any compression.
> >
> > That said, I'm open to contributing this to either pyarrow or avro if
> > there's interest.
> >
> > If the answer is "no" (as I suspect it is) and I don't contribute it now,
> > the package will be clearly identified as a fork of the Apache Avro
> project
> > and licensed Apache 2.0, so it should be easy to pull in once the
> > techniques are proven.
> >
> > * • **Tim Swast*
> > * • *Software Friendliness Engineer
> > * • *Google Cloud Developer Relations
> > * • *Seattle, WA, USA
> >
>
--
* • **Tim Swast*
* • *Software Friendliness Engineer
* • *Google Cloud Developer Relations
* • *Seattle, WA, USA
Re: Avro to Arrow?
Posted by Tim Swast <sw...@google.com.INVALID>.
Thanks for the advice, Wes.
Unfortunately, I am about 8 years out of practice for writing any C++
(which was part of the appeal of numba to me). Sounds like I should refresh
my skills. I like the idea of write one, have good performance everywhere.
On Tue, Jun 11, 2019 at 3:40 PM Wes McKinney <we...@gmail.com> wrote:
> Hi Tim,
>
> I'd ideally like to see the work done in the Arrow C++ library so that it
> can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB). This
> also means a larger labor pool of individuals to help improve and maintain
> the software. There was a stalled PR around this a time back (check out the
> Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
> might be more expedient to fork parts of Apache Avro and do all the
> development inside a single codebase.
>
> There's a lot of folks that can provide feedback should you choose to go
> down this route.
>
> Thanks
> Wes
>
> On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid> wrote:
>
> > Hi Arrow and Avro devs,
> >
> > I've been investigating some performance issues with the BigQuery Storage
> > API (https://github.com/googleapis/google-cloud-python/issues/7805), and
> > have identified that the vast majority of time is spent decoding Avro
> into
> > pandas dataframes.
> > <https://github.com/googleapis/google-cloud-python/issues/7805>
> > I've done some initial experiments by hand written parsers (inspired by
> > https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> > dramatic improvement in time spent parsing.
> >
> > I'm considering releasing this as a separate package for the following
> > reasons:
> >
> > - Code generation + Numba is a bit of an unproven technique for
> parsers,
> > so I'd like to treat this as an experiment rather than "the" package
> to
> > use
> > to parse Avro from Python.
> > - I don't need to handle the full Avro spec for this experiment.
> > Importantly, BQ Storage API only uses a schemaless reader (since the
> > schema
> > is output only once, and omitted for subsequent protobuf messages) and
> > doesn't use any compression.
> >
> > That said, I'm open to contributing this to either pyarrow or avro if
> > there's interest.
> >
> > If the answer is "no" (as I suspect it is) and I don't contribute it now,
> > the package will be clearly identified as a fork of the Apache Avro
> project
> > and licensed Apache 2.0, so it should be easy to pull in once the
> > techniques are proven.
> >
> > * • **Tim Swast*
> > * • *Software Friendliness Engineer
> > * • *Google Cloud Developer Relations
> > * • *Seattle, WA, USA
> >
>
--
* • **Tim Swast*
* • *Software Friendliness Engineer
* • *Google Cloud Developer Relations
* • *Seattle, WA, USA
Re: Avro to Arrow?
Posted by Wes McKinney <we...@gmail.com>.
Hi Tim,
I'd ideally like to see the work done in the Arrow C++ library so that it
can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB). This
also means a larger labor pool of individuals to help improve and maintain
the software. There was a stalled PR around this a time back (check out the
Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
might be more expedient to fork parts of Apache Avro and do all the
development inside a single codebase.
There's a lot of folks that can provide feedback should you choose to go
down this route.
Thanks
Wes
On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid> wrote:
> Hi Arrow and Avro devs,
>
> I've been investigating some performance issues with the BigQuery Storage
> API (https://github.com/googleapis/google-cloud-python/issues/7805), and
> have identified that the vast majority of time is spent decoding Avro into
> pandas dataframes.
> <https://github.com/googleapis/google-cloud-python/issues/7805>
> I've done some initial experiments by hand written parsers (inspired by
> https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> dramatic improvement in time spent parsing.
>
> I'm considering releasing this as a separate package for the following
> reasons:
>
> - Code generation + Numba is a bit of an unproven technique for parsers,
> so I'd like to treat this as an experiment rather than "the" package to
> use
> to parse Avro from Python.
> - I don't need to handle the full Avro spec for this experiment.
> Importantly, BQ Storage API only uses a schemaless reader (since the
> schema
> is output only once, and omitted for subsequent protobuf messages) and
> doesn't use any compression.
>
> That said, I'm open to contributing this to either pyarrow or avro if
> there's interest.
>
> If the answer is "no" (as I suspect it is) and I don't contribute it now,
> the package will be clearly identified as a fork of the Apache Avro project
> and licensed Apache 2.0, so it should be easy to pull in once the
> techniques are proven.
>
> * • **Tim Swast*
> * • *Software Friendliness Engineer
> * • *Google Cloud Developer Relations
> * • *Seattle, WA, USA
>
Re: Avro to Arrow?
Posted by Wes McKinney <we...@gmail.com>.
Hi Tim,
I'd ideally like to see the work done in the Arrow C++ library so that it
can be utilized by all the C++ "binders" (Python, R, C, Ruby, MATLAB). This
also means a larger labor pool of individuals to help improve and maintain
the software. There was a stalled PR around this a time back (check out the
Arrow Closed PR queue) that got stuck on some limitations in avro-c. It
might be more expedient to fork parts of Apache Avro and do all the
development inside a single codebase.
There's a lot of folks that can provide feedback should you choose to go
down this route.
Thanks
Wes
On Tue, Jun 11, 2019, 4:53 PM Tim Swast <sw...@google.com.invalid> wrote:
> Hi Arrow and Avro devs,
>
> I've been investigating some performance issues with the BigQuery Storage
> API (https://github.com/googleapis/google-cloud-python/issues/7805), and
> have identified that the vast majority of time is spent decoding Avro into
> pandas dataframes.
> <https://github.com/googleapis/google-cloud-python/issues/7805>
> I've done some initial experiments by hand written parsers (inspired by
> https://techblog.rtbhouse.com/2017/04/18/fast-avro/) and have seen a
> dramatic improvement in time spent parsing.
>
> I'm considering releasing this as a separate package for the following
> reasons:
>
> - Code generation + Numba is a bit of an unproven technique for parsers,
> so I'd like to treat this as an experiment rather than "the" package to
> use
> to parse Avro from Python.
> - I don't need to handle the full Avro spec for this experiment.
> Importantly, BQ Storage API only uses a schemaless reader (since the
> schema
> is output only once, and omitted for subsequent protobuf messages) and
> doesn't use any compression.
>
> That said, I'm open to contributing this to either pyarrow or avro if
> there's interest.
>
> If the answer is "no" (as I suspect it is) and I don't contribute it now,
> the package will be clearly identified as a fork of the Apache Avro project
> and licensed Apache 2.0, so it should be easy to pull in once the
> techniques are proven.
>
> * • **Tim Swast*
> * • *Software Friendliness Engineer
> * • *Google Cloud Developer Relations
> * • *Seattle, WA, USA
>