You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Benson Muite <be...@emailplus.org> on 2022/04/27 11:04:53 UTC
Re: Arrow sync call April 27 at 12:00 US/Eastern, 16:00 UTC
Hi,
Can host if required, though the timing is not ideal for me. It may be
helpful to vary the timing in future.
Benson
On 4/25/22 2:49 PM, David Li wrote:
> Following up here:
>
>> N.B. The Voltron Data folks have a scheduling conflict on 4/27 and will not be able to host the fortnightly sync call. Is anyone available to run the meeting that day?
>
> Is anyone available to run the sync call this Wednesday?
>
> On Wed, Apr 13, 2022, at 12:59, David Li wrote:
>> Attendees:
>>
>> - David Li
>> - Eduardo Ponce
>> - Gavin Ray
>> - Ian Cook
>> - James Duong
>> - Matthew Topol
>> - Nic
>> - Niranda
>> - Raul Cumplido
>> - Rok
>> - Weston Pace
>> - Will Jones
>>
>> N.B. The Voltron Data folks have a scheduling conflict on 4/27 and will
>> not be able to host the fortnightly sync call. Is anyone available to
>> run the meeting that day?
>>
>> Agenda:
>>
>> 8.0.0 Release: targeting 4/21, please try to get PRs wrapped up in the
>> next ~1-2 weeks. See the ML post [1] for details, including a wiki page
>> listing outstanding issues. In particular, there are some Go PRs that
>> could use attention from an interested Go developer [2], as well as
>> some temporal kernel PRs that could use a review [3].
>>
>> Arrow C++ Compute Engine: Weston gave a status update;
>> APIs/documentation has been improved for users, though likely most will
>> use it through an API like Substrait; basic Substrait support has been
>> added with forthcoming improvements; more tooling to measure
>> performance is being worked on; general kernel execution overhead is
>> being addressed with an eye towards running smaller batches through the
>> engine. An asof join implementation is being worked on, and Go is
>> working towards Substrait bindings to be able to bind to the C++ engine.
>>
>> Kernel vectorization/SIMD: Eduardo has been looking at making some of
>> the primitive kernels (e.g. arithmetic) more easily autovectorized by
>> the compiler, testing a variety of approaches. See related discussion
>> [4]. We do not have benchmarks to evaluate compiler performance in this
>> regard generally, but we have manually inspected some compiler output
>> and found that not all compilers manage to do this with the current
>> kernel implementations. We also don't have a holistic way to evaluate
>> this going forward, nor do we have a sense for current benchmark
>> coverage, though possibly we could generate benchmarks. However, it was
>> pointed out that general engine performance is likely more important,
>> and that current profiling indicates kernels are not yet a bottleneck,
>> though there may be low-hanging fruit here.
>>
>> Flight/Flight SQL: we discussed the barriers to Flight SQL support in
>> Go; Flight SQL heavily uses union types which are not yet implemented.
>> A further proposal [5] has been submitted to extend the type metadata,
>> please take a look for those interested. The GetXdbcTypeInfo proposal
>> was merged, and the inline data proposal is still outstanding (but
>> probably ready to have a vote).
>>
>> IPC/Format: it was asked if there's an IPC structure for serializing a
>> single array to reduce overhead. Current APIs likely suffice but
>> Niranda may submit a separate discussion to explain further.
>>
>> [1]: https://lists.apache.org/thread/zk8hhynvy0bqvqpxk0868n5g0nmzbzbn
>> [2]: https://github.com/apache/arrow/pull/12158
>> [3]: https://github.com/apache/arrow/pull/12657
>> [4]: https://lists.apache.org/thread/8o7k4dt23chx3gn13rwkms38syyms489
>> [5]: https://lists.apache.org/thread/thvn89wg29gyctwycx2zjr4vvm2g80o6
>>
>> On Tue, Apr 12, 2022, at 16:17, Ian Cook wrote:
>>> Hi all,
>>>
>>> Our biweekly sync call is tomorrow at 12:00 noon Eastern time.
>>>
>>> The Zoom meeting URL for this and other biweekly Arrow sync calls is:
>>> https://zoom.us/j/87649033008?pwd=SitsRHluQStlREM0TjJVYkRibVZsUT09
>>>
>>> Alternatively, enter this information into the Zoom website or app to
>>> join the call:
>>> Meeting ID: 876 4903 3008
>>> Passcode: 958092
>>>
>>> Thanks,
>>> Ian
Re: Arrow sync call April 27 at 12:00 US/Eastern, 16:00 UTC
Posted by Benson Muite <be...@emailplus.org>.
Attendees:
Ian Joiner
Matthew Topol
Benson Muite
Discussion points:
1) New book on Arrow - covers C++, Python and Go, out in June
2) Building ORC bindings in R would be useful, extensions to parallel R?
3) Comparing ORC and Parquet for IO
4) IO optimization vs SIMD optimization - Parquet seems well optimized,
so SIMD would be more helpful, but SIMD requires some care from Go.
5) Substrait would be great to use from Go if it is developed more fully.
6) A developer guide to developing Arrow on the cloud maybe useful
Re: Arrow sync call April 27 at 12:00 US/Eastern, 16:00 UTC
Posted by Ian Cook <ia...@ursacomputing.com>.
Thanks Benson!
The Zoom meeting URL for this and other biweekly Arrow sync calls is:
https://zoom.us/j/87649033008?pwd=SitsRHluQStlREM0TjJVYkRibVZsUT09
Alternatively, enter this information into the Zoom website or app to
join the call:
Meeting ID: 876 4903 3008
Passcode: 958092
The Zoom meeting will start when the first attendee joins, even if
there is no one from Voltron Data present.
Ian
On Wed, Apr 27, 2022 at 9:45 AM David Li <li...@apache.org> wrote:
>
> Thanks Benson. If you are able to take notes this week that would be much appreciated.
>
> And thanks Joris for the clarification.
>
> On Wed, Apr 27, 2022, at 09:34, Joris Van den Bossche wrote:
> > As a small clarification: the zoom meeting link itself should still work
> > for anyone to join, it's only there is no one from Voltron Data to lead the
> > meeting / take notes (so I also won't be present today).
> >
> > Joris
> >
> > On Wed, 27 Apr 2022 at 13:05, Benson Muite <be...@emailplus.org>
> > wrote:
> >
> >> Hi,
> >>
> >> Can host if required, though the timing is not ideal for me. It may be
> >> helpful to vary the timing in future.
> >>
> >> Benson
> >>
> >> On 4/25/22 2:49 PM, David Li wrote:
> >> > Following up here:
> >> >
> >> >> N.B. The Voltron Data folks have a scheduling conflict on 4/27 and will
> >> not be able to host the fortnightly sync call. Is anyone available to run
> >> the meeting that day?
> >> >
> >> > Is anyone available to run the sync call this Wednesday?
> >> >
> >> > On Wed, Apr 13, 2022, at 12:59, David Li wrote:
> >> >> Attendees:
> >> >>
> >> >> - David Li
> >> >> - Eduardo Ponce
> >> >> - Gavin Ray
> >> >> - Ian Cook
> >> >> - James Duong
> >> >> - Matthew Topol
> >> >> - Nic
> >> >> - Niranda
> >> >> - Raul Cumplido
> >> >> - Rok
> >> >> - Weston Pace
> >> >> - Will Jones
> >> >>
> >> >> N.B. The Voltron Data folks have a scheduling conflict on 4/27 and will
> >> >> not be able to host the fortnightly sync call. Is anyone available to
> >> >> run the meeting that day?
> >> >>
> >> >> Agenda:
> >> >>
> >> >> 8.0.0 Release: targeting 4/21, please try to get PRs wrapped up in the
> >> >> next ~1-2 weeks. See the ML post [1] for details, including a wiki page
> >> >> listing outstanding issues. In particular, there are some Go PRs that
> >> >> could use attention from an interested Go developer [2], as well as
> >> >> some temporal kernel PRs that could use a review [3].
> >> >>
> >> >> Arrow C++ Compute Engine: Weston gave a status update;
> >> >> APIs/documentation has been improved for users, though likely most will
> >> >> use it through an API like Substrait; basic Substrait support has been
> >> >> added with forthcoming improvements; more tooling to measure
> >> >> performance is being worked on; general kernel execution overhead is
> >> >> being addressed with an eye towards running smaller batches through the
> >> >> engine. An asof join implementation is being worked on, and Go is
> >> >> working towards Substrait bindings to be able to bind to the C++ engine.
> >> >>
> >> >> Kernel vectorization/SIMD: Eduardo has been looking at making some of
> >> >> the primitive kernels (e.g. arithmetic) more easily autovectorized by
> >> >> the compiler, testing a variety of approaches. See related discussion
> >> >> [4]. We do not have benchmarks to evaluate compiler performance in this
> >> >> regard generally, but we have manually inspected some compiler output
> >> >> and found that not all compilers manage to do this with the current
> >> >> kernel implementations. We also don't have a holistic way to evaluate
> >> >> this going forward, nor do we have a sense for current benchmark
> >> >> coverage, though possibly we could generate benchmarks. However, it was
> >> >> pointed out that general engine performance is likely more important,
> >> >> and that current profiling indicates kernels are not yet a bottleneck,
> >> >> though there may be low-hanging fruit here.
> >> >>
> >> >> Flight/Flight SQL: we discussed the barriers to Flight SQL support in
> >> >> Go; Flight SQL heavily uses union types which are not yet implemented.
> >> >> A further proposal [5] has been submitted to extend the type metadata,
> >> >> please take a look for those interested. The GetXdbcTypeInfo proposal
> >> >> was merged, and the inline data proposal is still outstanding (but
> >> >> probably ready to have a vote).
> >> >>
> >> >> IPC/Format: it was asked if there's an IPC structure for serializing a
> >> >> single array to reduce overhead. Current APIs likely suffice but
> >> >> Niranda may submit a separate discussion to explain further.
> >> >>
> >> >> [1]: https://lists.apache.org/thread/zk8hhynvy0bqvqpxk0868n5g0nmzbzbn
> >> >> [2]: https://github.com/apache/arrow/pull/12158
> >> >> [3]: https://github.com/apache/arrow/pull/12657
> >> >> [4]: https://lists.apache.org/thread/8o7k4dt23chx3gn13rwkms38syyms489
> >> >> [5]: https://lists.apache.org/thread/thvn89wg29gyctwycx2zjr4vvm2g80o6
> >> >>
> >> >> On Tue, Apr 12, 2022, at 16:17, Ian Cook wrote:
> >> >>> Hi all,
> >> >>>
> >> >>> Our biweekly sync call is tomorrow at 12:00 noon Eastern time.
> >> >>>
> >> >>> The Zoom meeting URL for this and other biweekly Arrow sync calls is:
> >> >>> https://zoom.us/j/87649033008?pwd=SitsRHluQStlREM0TjJVYkRibVZsUT09
> >> >>>
> >> >>> Alternatively, enter this information into the Zoom website or app to
> >> >>> join the call:
> >> >>> Meeting ID: 876 4903 3008
> >> >>> Passcode: 958092
> >> >>>
> >> >>> Thanks,
> >> >>> Ian
> >>
> >>
Re: Arrow sync call April 27 at 12:00 US/Eastern, 16:00 UTC
Posted by David Li <li...@apache.org>.
Thanks Benson. If you are able to take notes this week that would be much appreciated.
And thanks Joris for the clarification.
On Wed, Apr 27, 2022, at 09:34, Joris Van den Bossche wrote:
> As a small clarification: the zoom meeting link itself should still work
> for anyone to join, it's only there is no one from Voltron Data to lead the
> meeting / take notes (so I also won't be present today).
>
> Joris
>
> On Wed, 27 Apr 2022 at 13:05, Benson Muite <be...@emailplus.org>
> wrote:
>
>> Hi,
>>
>> Can host if required, though the timing is not ideal for me. It may be
>> helpful to vary the timing in future.
>>
>> Benson
>>
>> On 4/25/22 2:49 PM, David Li wrote:
>> > Following up here:
>> >
>> >> N.B. The Voltron Data folks have a scheduling conflict on 4/27 and will
>> not be able to host the fortnightly sync call. Is anyone available to run
>> the meeting that day?
>> >
>> > Is anyone available to run the sync call this Wednesday?
>> >
>> > On Wed, Apr 13, 2022, at 12:59, David Li wrote:
>> >> Attendees:
>> >>
>> >> - David Li
>> >> - Eduardo Ponce
>> >> - Gavin Ray
>> >> - Ian Cook
>> >> - James Duong
>> >> - Matthew Topol
>> >> - Nic
>> >> - Niranda
>> >> - Raul Cumplido
>> >> - Rok
>> >> - Weston Pace
>> >> - Will Jones
>> >>
>> >> N.B. The Voltron Data folks have a scheduling conflict on 4/27 and will
>> >> not be able to host the fortnightly sync call. Is anyone available to
>> >> run the meeting that day?
>> >>
>> >> Agenda:
>> >>
>> >> 8.0.0 Release: targeting 4/21, please try to get PRs wrapped up in the
>> >> next ~1-2 weeks. See the ML post [1] for details, including a wiki page
>> >> listing outstanding issues. In particular, there are some Go PRs that
>> >> could use attention from an interested Go developer [2], as well as
>> >> some temporal kernel PRs that could use a review [3].
>> >>
>> >> Arrow C++ Compute Engine: Weston gave a status update;
>> >> APIs/documentation has been improved for users, though likely most will
>> >> use it through an API like Substrait; basic Substrait support has been
>> >> added with forthcoming improvements; more tooling to measure
>> >> performance is being worked on; general kernel execution overhead is
>> >> being addressed with an eye towards running smaller batches through the
>> >> engine. An asof join implementation is being worked on, and Go is
>> >> working towards Substrait bindings to be able to bind to the C++ engine.
>> >>
>> >> Kernel vectorization/SIMD: Eduardo has been looking at making some of
>> >> the primitive kernels (e.g. arithmetic) more easily autovectorized by
>> >> the compiler, testing a variety of approaches. See related discussion
>> >> [4]. We do not have benchmarks to evaluate compiler performance in this
>> >> regard generally, but we have manually inspected some compiler output
>> >> and found that not all compilers manage to do this with the current
>> >> kernel implementations. We also don't have a holistic way to evaluate
>> >> this going forward, nor do we have a sense for current benchmark
>> >> coverage, though possibly we could generate benchmarks. However, it was
>> >> pointed out that general engine performance is likely more important,
>> >> and that current profiling indicates kernels are not yet a bottleneck,
>> >> though there may be low-hanging fruit here.
>> >>
>> >> Flight/Flight SQL: we discussed the barriers to Flight SQL support in
>> >> Go; Flight SQL heavily uses union types which are not yet implemented.
>> >> A further proposal [5] has been submitted to extend the type metadata,
>> >> please take a look for those interested. The GetXdbcTypeInfo proposal
>> >> was merged, and the inline data proposal is still outstanding (but
>> >> probably ready to have a vote).
>> >>
>> >> IPC/Format: it was asked if there's an IPC structure for serializing a
>> >> single array to reduce overhead. Current APIs likely suffice but
>> >> Niranda may submit a separate discussion to explain further.
>> >>
>> >> [1]: https://lists.apache.org/thread/zk8hhynvy0bqvqpxk0868n5g0nmzbzbn
>> >> [2]: https://github.com/apache/arrow/pull/12158
>> >> [3]: https://github.com/apache/arrow/pull/12657
>> >> [4]: https://lists.apache.org/thread/8o7k4dt23chx3gn13rwkms38syyms489
>> >> [5]: https://lists.apache.org/thread/thvn89wg29gyctwycx2zjr4vvm2g80o6
>> >>
>> >> On Tue, Apr 12, 2022, at 16:17, Ian Cook wrote:
>> >>> Hi all,
>> >>>
>> >>> Our biweekly sync call is tomorrow at 12:00 noon Eastern time.
>> >>>
>> >>> The Zoom meeting URL for this and other biweekly Arrow sync calls is:
>> >>> https://zoom.us/j/87649033008?pwd=SitsRHluQStlREM0TjJVYkRibVZsUT09
>> >>>
>> >>> Alternatively, enter this information into the Zoom website or app to
>> >>> join the call:
>> >>> Meeting ID: 876 4903 3008
>> >>> Passcode: 958092
>> >>>
>> >>> Thanks,
>> >>> Ian
>>
>>
Re: Arrow sync call April 27 at 12:00 US/Eastern, 16:00 UTC
Posted by Joris Van den Bossche <jo...@gmail.com>.
As a small clarification: the zoom meeting link itself should still work
for anyone to join, it's only there is no one from Voltron Data to lead the
meeting / take notes (so I also won't be present today).
Joris
On Wed, 27 Apr 2022 at 13:05, Benson Muite <be...@emailplus.org>
wrote:
> Hi,
>
> Can host if required, though the timing is not ideal for me. It may be
> helpful to vary the timing in future.
>
> Benson
>
> On 4/25/22 2:49 PM, David Li wrote:
> > Following up here:
> >
> >> N.B. The Voltron Data folks have a scheduling conflict on 4/27 and will
> not be able to host the fortnightly sync call. Is anyone available to run
> the meeting that day?
> >
> > Is anyone available to run the sync call this Wednesday?
> >
> > On Wed, Apr 13, 2022, at 12:59, David Li wrote:
> >> Attendees:
> >>
> >> - David Li
> >> - Eduardo Ponce
> >> - Gavin Ray
> >> - Ian Cook
> >> - James Duong
> >> - Matthew Topol
> >> - Nic
> >> - Niranda
> >> - Raul Cumplido
> >> - Rok
> >> - Weston Pace
> >> - Will Jones
> >>
> >> N.B. The Voltron Data folks have a scheduling conflict on 4/27 and will
> >> not be able to host the fortnightly sync call. Is anyone available to
> >> run the meeting that day?
> >>
> >> Agenda:
> >>
> >> 8.0.0 Release: targeting 4/21, please try to get PRs wrapped up in the
> >> next ~1-2 weeks. See the ML post [1] for details, including a wiki page
> >> listing outstanding issues. In particular, there are some Go PRs that
> >> could use attention from an interested Go developer [2], as well as
> >> some temporal kernel PRs that could use a review [3].
> >>
> >> Arrow C++ Compute Engine: Weston gave a status update;
> >> APIs/documentation has been improved for users, though likely most will
> >> use it through an API like Substrait; basic Substrait support has been
> >> added with forthcoming improvements; more tooling to measure
> >> performance is being worked on; general kernel execution overhead is
> >> being addressed with an eye towards running smaller batches through the
> >> engine. An asof join implementation is being worked on, and Go is
> >> working towards Substrait bindings to be able to bind to the C++ engine.
> >>
> >> Kernel vectorization/SIMD: Eduardo has been looking at making some of
> >> the primitive kernels (e.g. arithmetic) more easily autovectorized by
> >> the compiler, testing a variety of approaches. See related discussion
> >> [4]. We do not have benchmarks to evaluate compiler performance in this
> >> regard generally, but we have manually inspected some compiler output
> >> and found that not all compilers manage to do this with the current
> >> kernel implementations. We also don't have a holistic way to evaluate
> >> this going forward, nor do we have a sense for current benchmark
> >> coverage, though possibly we could generate benchmarks. However, it was
> >> pointed out that general engine performance is likely more important,
> >> and that current profiling indicates kernels are not yet a bottleneck,
> >> though there may be low-hanging fruit here.
> >>
> >> Flight/Flight SQL: we discussed the barriers to Flight SQL support in
> >> Go; Flight SQL heavily uses union types which are not yet implemented.
> >> A further proposal [5] has been submitted to extend the type metadata,
> >> please take a look for those interested. The GetXdbcTypeInfo proposal
> >> was merged, and the inline data proposal is still outstanding (but
> >> probably ready to have a vote).
> >>
> >> IPC/Format: it was asked if there's an IPC structure for serializing a
> >> single array to reduce overhead. Current APIs likely suffice but
> >> Niranda may submit a separate discussion to explain further.
> >>
> >> [1]: https://lists.apache.org/thread/zk8hhynvy0bqvqpxk0868n5g0nmzbzbn
> >> [2]: https://github.com/apache/arrow/pull/12158
> >> [3]: https://github.com/apache/arrow/pull/12657
> >> [4]: https://lists.apache.org/thread/8o7k4dt23chx3gn13rwkms38syyms489
> >> [5]: https://lists.apache.org/thread/thvn89wg29gyctwycx2zjr4vvm2g80o6
> >>
> >> On Tue, Apr 12, 2022, at 16:17, Ian Cook wrote:
> >>> Hi all,
> >>>
> >>> Our biweekly sync call is tomorrow at 12:00 noon Eastern time.
> >>>
> >>> The Zoom meeting URL for this and other biweekly Arrow sync calls is:
> >>> https://zoom.us/j/87649033008?pwd=SitsRHluQStlREM0TjJVYkRibVZsUT09
> >>>
> >>> Alternatively, enter this information into the Zoom website or app to
> >>> join the call:
> >>> Meeting ID: 876 4903 3008
> >>> Passcode: 958092
> >>>
> >>> Thanks,
> >>> Ian
>
>