You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by "Du, Frank" <fr...@intel.com> on 2020/04/29 06:25:56 UTC
Vendor direction for SSE operations on Spaced Decode/Encode
Hi All,
Currently we are working on SIMD chance to improves speed of DecodeSpaced on Jira<https://issues.apache.org/jira/browse/PARQUET-1841>, the path for AVX512 is identified with the helper of vpcompressd/vpexpandd instructions. I has uploaded the PR<https://github.com/apache/arrow/pull/7029> for this path and other SIMD unittest, benchmark things.
For the SSE chance, thanks to help from Micah Kornfield, the path can be achieved by a lookup table, method referenced from https://github.com/lemire/simdprune. I has a draft implementation locally which demonstrate double speed on epi32 path and 50% improvements on epi64.
The issue is how to approach the reference, one way is vendor the whole simdprune library while another choice is just introducing the lookup table/script we really used. I personally like the later one as we need expand the table to support both encode/decode, simdprune only has encode table, also we need a new epi64 table to support the double type. Thoughts are welcomeš
Thanks,
Frank
Re: Vendor direction for SSE operations on Spaced Decode/Encode
Posted by Wes McKinney <we...@gmail.com>.
You can see how we've done bookkeeping for some other vendored code from
Daniel Lemire
https://github.com/apache/arrow/blob/master/LICENSE.txt#L313
On Wed, Apr 29, 2020 at 3:05 AM Antoine Pitrou <an...@python.org> wrote:
>
> Hi Frank,
>
> Vendoring just the required lookup table sounds fine to me.
>
> Regards
>
> Antoine.
>
>
>
> Le 29/04/2020 Ć 08:25, Du, Frank a Ć©crit :
> > Hi All,
> >
> > Currently we are working on SIMD chance to improves speed of
> DecodeSpaced on Jira<https://issues.apache.org/jira/browse/PARQUET-1841>,
> the path for AVX512 is identified with the helper of vpcompressd/vpexpandd
> instructions. I has uploaded the PR<
> https://github.com/apache/arrow/pull/7029> for this path and other SIMD
> unittest, benchmark things.
> >
> > For the SSE chance, thanks to help from Micah Kornfield, the path can be
> achieved by a lookup table, method referenced from
> https://github.com/lemire/simdprune. I has a draft implementation locally
> which demonstrate double speed on epi32 path and 50% improvements on epi64.
> >
> > The issue is how to approach the reference, one way is vendor the whole
> simdprune library while another choice is just introducing the lookup
> table/script we really used. I personally like the later one as we need
> expand the table to support both encode/decode, simdprune only has encode
> table, also we need a new epi64 table to support the double type. Thoughts
> are welcomeš
> >
> > Thanks,
> > Frank
> >
>
Re: Vendor direction for SSE operations on Spaced Decode/Encode
Posted by Antoine Pitrou <an...@python.org>.
Hi Frank,
Vendoring just the required lookup table sounds fine to me.
Regards
Antoine.
Le 29/04/2020 Ć 08:25, Du, Frank a Ć©critĀ :
> Hi All,
>
> Currently we are working on SIMD chance to improves speed of DecodeSpaced on Jira<https://issues.apache.org/jira/browse/PARQUET-1841>, the path for AVX512 is identified with the helper of vpcompressd/vpexpandd instructions. I has uploaded the PR<https://github.com/apache/arrow/pull/7029> for this path and other SIMD unittest, benchmark things.
>
> For the SSE chance, thanks to help from Micah Kornfield, the path can be achieved by a lookup table, method referenced from https://github.com/lemire/simdprune. I has a draft implementation locally which demonstrate double speed on epi32 path and 50% improvements on epi64.
>
> The issue is how to approach the reference, one way is vendor the whole simdprune library while another choice is just introducing the lookup table/script we really used. I personally like the later one as we need expand the table to support both encode/decode, simdprune only has encode table, also we need a new epi64 table to support the double type. Thoughts are welcomeš
>
> Thanks,
> Frank
>