You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AntoinePrv (via GitHub)" <gi...@apache.org> on 2025/10/29 14:23:25 UTC
[PR] unpack with shuffle algorithm [arrow]
AntoinePrv opened a new pull request, #47994:
URL: https://github.com/apache/arrow/pull/47994
### Rationale for this change
WIP
### What changes are included in this PR?
Currently also includes changes required for local benchmarks.
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2797688469
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
Review Comment:
Byte-aligned. We never mention SIMD aligned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793255084
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
Review Comment:
Ok. Then just replace "in the SSE family" with "in the AVX2 family"? :-)
(i.e. that still excludes AVX-512 and AVX10)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783129267
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
Review Comment:
Yes some will be undefined. It usually set the corresponding value to zero, though we do not rely on it here. Bytes that are left undefined are not used (they are eventually masked out).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921026744
Thanks for your patience. Conbench analyzed the 3 benchmarking runs that have been run so far on PR commit 97d87e8a5ae6f9d1e337163b20f98a2aa1974c6e.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64011937917) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921053077
Not sure there's anything else useful we can do with the Windows R CI that isn't a massive pain - probably worth just seeing what comes out of the sanitiser jobs first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3925581059
The JNI failures are unrelated, but the R Windows failure is the same crash as before, presumably :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783058104
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
Review Comment:
Yes it should. It is "slow" WIP. In arrow we only care about a few cases but in xsimd we need to implement the API for all architectures so it is more work to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3915841768
Unfortunately the "AMD64 Windows R release" failure looks related to this PR: it's deterministic (I've restarted it twice), it doesn't occur on git main, and the test where it fails/crashes uses Parquet.
https://github.com/apache/arrow/blob/ebaaf07adbd302e95e393b5b77d78c1c97ea3b70/r/tests/testthat/test-dplyr-summarize.R#L260-L271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3922059705
Rebased to see if GH-49326 helps.
@pitrou would you mind relaunching the extra tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793246448
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
Review Comment:
In xsimd, there is `avx2+fma3` that is currently implemented as inheriting from `avx2`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3597357073
@ursabot please benchmark lang=C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3586939442
@pitrou apart from R-lint, this is looking pretty good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3586990419
@ursabot please benchmark lang=C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "voltrondatabot (via GitHub)" <gi...@apache.org>.
voltrondatabot commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3589470428
Benchmark runs are scheduled for commit dd3ec0d692e4f409bd73952de9bab20d8c97c226. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3590068536
Thanks for your patience. Conbench analyzed the 4 benchmarking runs that have been run so far on PR commit dd3ec0d692e4f409bd73952de9bab20d8c97c226.
There were 19 benchmark results indicating a performance regression:
- Pull Request Run on `amd64-c6a-4xlarge-linux` at [2025-11-28 15:12:47Z](https://conbench.ursa.dev/compare/runs/2d25bf63d48b4d54abb5aa4718492357...387353b7f5284f18a3956b7291b44881/)
- [`BM_UnpackUint32` (C++) with params=DynamicUnaligned/20/64, source=cpp-micro, suite=arrow-bpacking-benchmark](https://conbench.ursa.dev/compare/benchmarks/06927c1eff1f74aa8000d7c3337b60e7...06929bc7f23c766c8000e4298a5fc237)
- [`BM_DeltaLengthDecodingByteArray` (C++) with params=max-string-length:8/batch-size:2048, source=cpp-micro, suite=parquet-encoding-benchmark](https://conbench.ursa.dev/compare/benchmarks/06927c1b0b8974e78000f673802284b0...06929bc5d25b71cc8000b7015d9e7e05)
- and 17 more (see the report linked below)
The [full Conbench report](https://github.com/apache/arrow/runs/56655213639) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783049368
##########
cpp/src/arrow/util/bpacking_dispatch_internal.h:
##########
@@ -188,297 +202,431 @@ void unpack_width(const uint8_t* in, UnpackedUInt* out, int batch_size, int bit_
return unpack_full(in, out, batch_size);
} else {
using UnpackerForWidth = Unpacker<UnpackedUInt, kPackedBitWidth>;
+ // Number of values extracted by one iteration of the kernel
constexpr auto kValuesUnpacked = UnpackerForWidth::kValuesUnpacked;
+ // Number of bytes read, but not necessarily unpacked, by one iteration of the
+ // kernel. This constant prevent reading past buffer end.
+ constexpr auto kBytesRead = UnpackerForWidth::kBytesRead;
+
+ if constexpr (kValuesUnpacked > 0) {
+ const uint8_t* in_last = in_end - kBytesRead;
+ // Running the optimized kernel for batch extraction
+ while ((batch_size >= kValuesUnpacked) && (in <= in_last)) {
Review Comment:
Do you think it would change? All that code is header only so it all end up in the same compilation unit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2798027800
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ static_assert(kOffset < sizeof(ToInt) / sizeof(Int));
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family (not AVX).
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the AVX family (e.g. AVX2+FMA3 as
+/// implemented in xsimd but not AVX512).
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
Review Comment:
Does it matter if it's not the same type as `kMask0` (due to integer promotion rules)?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ static_assert(kOffset < sizeof(ToInt) / sizeof(Int));
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family (not AVX).
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the AVX family (e.g. AVX2+FMA3 as
+/// implemented in xsimd but not AVX512).
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled uint8_t permutation indices on the first line and
+/// resulting swizzled register on the second.
+///
+/// | Idx 0 | Idx 1 | Idx 0 | Idx 1 || Idx 0 | Idx 1 | Idx 1 | Idx 2 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
Review Comment:
```suggestion
/// Masking (bitwise AND) uses the same mask for all uint32_t
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ static_assert(kOffset < sizeof(ToInt) / sizeof(Int));
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family (not AVX).
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the AVX family (e.g. AVX2+FMA3 as
+/// implemented in xsimd but not AVX512).
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled uint8_t permutation indices on the first line and
+/// resulting swizzled register on the second.
+///
+/// | Idx 0 | Idx 1 | Idx 0 | Idx 1 || Idx 0 | Idx 1 | Idx 1 | Idx 2 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
+/// We therefore cannot end here and reapply the same kernel on the same input.
+/// Instead, we run another similar iteration to unpack E, F, G, H, this time taking into
+/// account that we have an extra offset of 12 bits (this will change all the constants
+/// used for swizzle and shift).
+///
+/// Note that in this example, we could have swizzled more than two values in each slot.
+/// In practice, there may be situations where a more complex algorithm could fit more
+/// shifts per swizzles, but that tend to not be the case when the SIMD register size
+/// increases.
+///
+/// @see KernelShape
+/// @see MediumKernelPlan
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = MediumKernelPlan<KerTraits, kOptions>::Build();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = decltype(kPlan)::Traits;
+ using unpacked_type = Traits::unpacked_type;
+ using uint_type = Traits::uint_type;
+ using simd_batch = Traits::simd_batch;
+ using simd_bytes = Traits::simd_bytes;
+ using arch_type = Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shift());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ safe_load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+/// Compile time dimensions of a Large kernel.
+///
+/// @see LargeKernel for explanation of the algorithm.
+struct LargeKernelPlanSize {
+ int reads_per_kernel_;
+ int unpacked_per_kernel_;
+
+ constexpr static LargeKernelPlanSize Build(const KernelShape& shape) {
+ const auto unpacked_per_kernel = std::lcm(shape.unpacked_per_simd(), 8);
+ const auto packed_bits_per_kernel = unpacked_per_kernel * shape.packed_bit_size();
+ const auto reads_per_kernel =
+ bit_util::CeilDiv(packed_bits_per_kernel, shape.simd_bit_size());
+
+ return {
+ .reads_per_kernel_ = static_cast<int>(reads_per_kernel),
+ .unpacked_per_kernel_ = unpacked_per_kernel,
+ };
+ }
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int unpacked_per_read() const {
+ return unpacked_per_kernel_ / reads_per_kernel_;
+ }
+ constexpr int unpacked_per_kernel() const { return unpacked_per_kernel_; }
+};
+
+/// Compile time constants of a Large kernel.
+///
+/// This contains most of the information about the large kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see LargeKernel for an explanation of the algorithm.
+template <typename KerTraits>
+struct LargeKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = LargeKernelPlanSize::Build(kShape);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shifts, kPlanSize.reads_per_kernel()>;
+
+ static constexpr LargeKernelPlan Build();
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kPlanSize.unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KerTraits>
+constexpr auto LargeKernelPlan<KerTraits>::Build() -> LargeKernelPlan<KerTraits> {
+ using Plan = LargeKernelPlan<KerTraits>;
+ using uint_type = Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ plan.low_swizzles.at(r) = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ // const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ // const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ const auto low_swizzle_val = packed_start_byte_in_read + b;
+ // Ensure we do not override previously defined values
+ assert(plan.low_swizzles.at(r).at(idx) == kUndefined);
+ assert(low_swizzle_val < kShape.simd_byte_size());
+
+ // Since kOverBytes is overly pessimistic, we may read an extra unneeded byte.
+ // On the last unpacked integer in the SIMD register, it may produce an out of
+ // bound swizzle. It would act the same as ``kUndefined`` but we make this
+ // explicit here.
+ // We need to stay pessimistic with ``kOverBytes`` to avoid having a mix of right
+ // and left shifts in the high swizzle.
+ auto high_swizzle_val = low_swizzle_val + kOverBytes;
+ if (high_swizzle_val >= kShape.simd_byte_size()) {
+ high_swizzle_val = kUndefined;
+ }
+ // Ensure we do not override previously defined values
+ assert(plan.high_swizzles.at(r).at(idx) == kUndefined);
+
+ plan.low_swizzles.at(r).at(idx) = static_cast<uint8_t>(low_swizzle_val);
+ plan.high_swizzles.at(r).at(idx) = static_cast<uint8_t>(high_swizzle_val);
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes create a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ assert(packed_start_bit % 8 == 0);
+
+ plan.mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for large sized inputs, where packed values spread over more bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 13 bits, extracted to
+/// uint16_t, using a fictive 32 bits SIMD size.
+/// The input register is as follow, where 13 consecutive letters represent the 13 bits of
+/// a packed value in little endian representation.
+///
+/// |AAAAAAAA|AAAAABBB||BBBBBBBB|BBCCCCCC|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 32/16 = 2 unpacked values per output register.
+/// Since 13-bit values can spread over 3 bytes (at worst bit alignment) but we're
+/// unpacking to uint16_t (2 bytes).
+///
+/// The core idea is to perform two separate swizzles: one for the "low" bits and one
+/// for the "high" bits of each value, then combine them with shifts and OR operations.
+///
+/// | Low bits from A || Low bits from B |
+/// low: |AAAAAAAA|AAAAABBB||AAAAABBB|BBBBBBBB|
+///
+/// | High bits frm A || High bits frm B |
+/// high: |AAAAABBB|BBBBBBBB||BBBBBBBB|BBCCCCCC|
+///
+/// We then apply different shifts to align the values bits. The low values are
+/// right-shifted by the bit offset, and the high values are left-shifted to fill the gap.
+/// Note that because of the figure is little endian, right (>>) and left (<<) shift
+/// operators actually shift the bits representation in the opposite directions.
+///
+/// | Align A w. >> 0 || Align B w. >> 5 |
+/// low: |AAAAAAAA|AAAAABBB||BBBBBBBBB|BB00000|
+///
+/// | Align A w. << 8 || Align B w. << 3 |
+/// high: |00000000|AAAAABBB||000BBBBB|BBBBBCCC|
+///
+/// We then proceed to merge both inputs. The idea is to mask out irrelevant bits from the
+/// high and low parts and merge with bitwise OR. However, in practice we can merge first
+/// and apply the mask only once afterwards (because we shifted zeros in). Below is the
+/// bitwise OR of both parts:
+///
+/// |AAAAAAAA|AAAAABBB||BBBBBBBBB|BB00000|
Review Comment:
```suggestion
/// |AAAAAAAA|AAAAABBB||BBBBBBBB|BBB00000|
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ static_assert(kOffset < sizeof(ToInt) / sizeof(Int));
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family (not AVX).
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the AVX family (e.g. AVX2+FMA3 as
+/// implemented in xsimd but not AVX512).
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled uint8_t permutation indices on the first line and
+/// resulting swizzled register on the second.
+///
+/// | Idx 0 | Idx 1 | Idx 0 | Idx 1 || Idx 0 | Idx 1 | Idx 1 | Idx 2 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
+/// We therefore cannot end here and reapply the same kernel on the same input.
+/// Instead, we run another similar iteration to unpack E, F, G, H, this time taking into
+/// account that we have an extra offset of 12 bits (this will change all the constants
+/// used for swizzle and shift).
+///
+/// Note that in this example, we could have swizzled more than two values in each slot.
+/// In practice, there may be situations where a more complex algorithm could fit more
+/// shifts per swizzles, but that tend to not be the case when the SIMD register size
+/// increases.
+///
+/// @see KernelShape
+/// @see MediumKernelPlan
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = MediumKernelPlan<KerTraits, kOptions>::Build();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = decltype(kPlan)::Traits;
+ using unpacked_type = Traits::unpacked_type;
+ using uint_type = Traits::uint_type;
+ using simd_batch = Traits::simd_batch;
+ using simd_bytes = Traits::simd_bytes;
+ using arch_type = Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shift());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ safe_load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+/// Compile time dimensions of a Large kernel.
+///
+/// @see LargeKernel for explanation of the algorithm.
+struct LargeKernelPlanSize {
+ int reads_per_kernel_;
+ int unpacked_per_kernel_;
+
+ constexpr static LargeKernelPlanSize Build(const KernelShape& shape) {
+ const auto unpacked_per_kernel = std::lcm(shape.unpacked_per_simd(), 8);
+ const auto packed_bits_per_kernel = unpacked_per_kernel * shape.packed_bit_size();
+ const auto reads_per_kernel =
+ bit_util::CeilDiv(packed_bits_per_kernel, shape.simd_bit_size());
+
+ return {
+ .reads_per_kernel_ = static_cast<int>(reads_per_kernel),
+ .unpacked_per_kernel_ = unpacked_per_kernel,
+ };
+ }
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int unpacked_per_read() const {
+ return unpacked_per_kernel_ / reads_per_kernel_;
+ }
+ constexpr int unpacked_per_kernel() const { return unpacked_per_kernel_; }
+};
+
+/// Compile time constants of a Large kernel.
+///
+/// This contains most of the information about the large kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see LargeKernel for an explanation of the algorithm.
+template <typename KerTraits>
+struct LargeKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = LargeKernelPlanSize::Build(kShape);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shifts, kPlanSize.reads_per_kernel()>;
+
+ static constexpr LargeKernelPlan Build();
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kPlanSize.unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KerTraits>
+constexpr auto LargeKernelPlan<KerTraits>::Build() -> LargeKernelPlan<KerTraits> {
+ using Plan = LargeKernelPlan<KerTraits>;
+ using uint_type = Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ plan.low_swizzles.at(r) = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ // const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ // const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ const auto low_swizzle_val = packed_start_byte_in_read + b;
+ // Ensure we do not override previously defined values
+ assert(plan.low_swizzles.at(r).at(idx) == kUndefined);
+ assert(low_swizzle_val < kShape.simd_byte_size());
+
+ // Since kOverBytes is overly pessimistic, we may read an extra unneeded byte.
+ // On the last unpacked integer in the SIMD register, it may produce an out of
+ // bound swizzle. It would act the same as ``kUndefined`` but we make this
+ // explicit here.
+ // We need to stay pessimistic with ``kOverBytes`` to avoid having a mix of right
+ // and left shifts in the high swizzle.
+ auto high_swizzle_val = low_swizzle_val + kOverBytes;
+ if (high_swizzle_val >= kShape.simd_byte_size()) {
+ high_swizzle_val = kUndefined;
+ }
+ // Ensure we do not override previously defined values
+ assert(plan.high_swizzles.at(r).at(idx) == kUndefined);
+
+ plan.low_swizzles.at(r).at(idx) = static_cast<uint8_t>(low_swizzle_val);
+ plan.high_swizzles.at(r).at(idx) = static_cast<uint8_t>(high_swizzle_val);
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes create a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ assert(packed_start_bit % 8 == 0);
+
+ plan.mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for large sized inputs, where packed values spread over more bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 13 bits, extracted to
+/// uint16_t, using a fictive 32 bits SIMD size.
+/// The input register is as follow, where 13 consecutive letters represent the 13 bits of
+/// a packed value in little endian representation.
+///
+/// |AAAAAAAA|AAAAABBB||BBBBBBBB|BBCCCCCC|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 32/16 = 2 unpacked values per output register.
+/// Since 13-bit values can spread over 3 bytes (at worst bit alignment) but we're
+/// unpacking to uint16_t (2 bytes).
+///
+/// The core idea is to perform two separate swizzles: one for the "low" bits and one
+/// for the "high" bits of each value, then combine them with shifts and OR operations.
+///
+/// | Low bits from A || Low bits from B |
+/// low: |AAAAAAAA|AAAAABBB||AAAAABBB|BBBBBBBB|
+///
+/// | High bits frm A || High bits frm B |
+/// high: |AAAAABBB|BBBBBBBB||BBBBBBBB|BBCCCCCC|
+///
+/// We then apply different shifts to align the values bits. The low values are
+/// right-shifted by the bit offset, and the high values are left-shifted to fill the gap.
+/// Note that because of the figure is little endian, right (>>) and left (<<) shift
+/// operators actually shift the bits representation in the opposite directions.
+///
+/// | Align A w. >> 0 || Align B w. >> 5 |
+/// low: |AAAAAAAA|AAAAABBB||BBBBBBBBB|BB00000|
Review Comment:
```suggestion
/// low: |AAAAAAAA|AAAAABBB||BBBBBBBB|BBB00000|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3891802819
@ursabot please benchmark lang=C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "voltrondatabot (via GitHub)" <gi...@apache.org>.
voltrondatabot commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3597357493
Benchmark runs are scheduled for commit 408ef04ad96a9752654cfe54d4de6c7c2eef08cc. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783168609
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
Review Comment:
Unfortunately this is not exactly this. This check is "are we building for a target that includes SSE2 but is still part of the SSE family", that is up to `sse4.2`. So `TargetIsSseFamily` && `HasSse2`.
Could be `BetweenSse2AndSse42`? Ugly but descriptive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783602424
##########
cpp/src/arrow/util/bpacking_dispatch_internal.h:
##########
@@ -188,297 +202,431 @@ void unpack_width(const uint8_t* in, UnpackedUInt* out, int batch_size, int bit_
return unpack_full(in, out, batch_size);
} else {
using UnpackerForWidth = Unpacker<UnpackedUInt, kPackedBitWidth>;
+ // Number of values extracted by one iteration of the kernel
constexpr auto kValuesUnpacked = UnpackerForWidth::kValuesUnpacked;
+ // Number of bytes read, but not necessarily unpacked, by one iteration of the
+ // kernel. This constant prevent reading past buffer end.
+ constexpr auto kBytesRead = UnpackerForWidth::kBytesRead;
+
+ if constexpr (kValuesUnpacked > 0) {
+ const uint8_t* in_last = in_end - kBytesRead;
+ // Running the optimized kernel for batch extraction
+ while ((batch_size >= kValuesUnpacked) && (in <= in_last)) {
Review Comment:
I have no idea :) It could make a difference if the compiler decides not to inline the inner function perhaps.
In any case, let's ignore this for now, perhaps adding a TODO for later experimentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2775334793
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
Review Comment:
I put it at the top and renamed `BuildConstantArrayLike`, it's mostly that it verbose to get value_type and size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793543378
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
Review Comment:
The beginning of a new iteration. First iteration started with pointer on A (aligned) and extracted A, B, C, D.
However we cannot advance the pointer and repeat the kernel loop from here because E is unaligned compared to A.
We need to add another loop _inside_ the kernel that will process E, F, G, H with bit offsets. Then on I we are aligned again so we can exit and leave the outer loop call the kernel again with a pointer on I.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793338183
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
Review Comment:
This is a limitation of the current algorithm. When we swizzle, we assume the worst byte spread (here 2) for all values. It does not care what is the in the other swizzled bytes, only adds their own. This is why we are doing `sizeof(uint32_t) / max_spread = 4 / 2 = 2` shifts per swizzle and not 5 (from your next comment).
So below are all three steps one next to the other that determine what goes where
```
Val we want. | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
Byte idx for val: | Idx 0 | Idx 1 | Idx 0 | Idx 1 || Idx 0 | Idx 1 | Idx 1 | Idx 2 |
Result data: |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
```
See next comment for discussion the improvements to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921004782
Thanks for your patience. Conbench analyzed the 3 benchmarking runs that have been run so far on PR commit a3720d4696c4657812f10a8e015e9221f9d88a7c.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64011478322) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921098657
Thanks for your patience. Conbench analyzed the 2 benchmarking runs that have been run so far on PR commit b63857072c2e9da4cf179cde0479150690ee1932.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64013631552) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921122142
> Could these builds still be compiling with C++17?
No, just old compilers (clang 15 and AppleClang 17). For example "Make typename more optional" [appeared in clang 16](https://en.cppreference.com/w/cpp/compiler_support/20.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783144434
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
Review Comment:
In this algorithm it should not.
Biggest blocker is that `assert` is not `constexpr`. Alternatively we could make a utility function that throw an exception, guarded by `std::is_constant_evaluated`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783604749
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
Review Comment:
Ok, so perhaps just add a short comment to mention that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793246448
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
Review Comment:
In xsimd, there is `avx2+fma3` that is currently implemented as inheriting from `avx2`. Still I added extra clarifications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2797481752
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
Review Comment:
"Aligned" here means SIMD-aligned? Something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921070570
Thanks for your patience. Conbench analyzed the 2 benchmarking runs that have been run so far on PR commit b63857072c2e9da4cf179cde0479150690ee1932.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64012995359) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3925570727
> Ok, I can add the `typename` again, but what about the `<concepts>`? Should I revert using them too?
Yes, unfortunately for now. Or at least the ones that were not found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3914915031
Revision: b63857072c2e9da4cf179cde0479150690ee1932
Submitted crossbow builds: [ursacomputing/crossbow @ actions-158ce97c66](https://github.com/ursacomputing/crossbow/branches/all?query=actions-158ce97c66)
|Task|Status|
|----|------|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714790/job/63872922205)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713477/job/63872916685)|
|example-cpp-tutorial|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714530/job/63872921350)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714004/job/63872919446)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713470/job/63872916628)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713875/job/63872919053)|
|test-debian-13-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713144/job/63872916215)|
|test-debian-13-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713607/job/63872917343)|
|test-debian-experimental-cpp-gcc-15|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714234/job/63872920348)|
|test-fedora-42-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713251/job/63872916762)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714959/job/63872922623)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713113/job/63872916128)|
|test-ubuntu-22.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713802/job/63872918279)|
|test-ubuntu-22.04-cpp-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713326/job/63872916644)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714295/job/63872920390)|
|test-ubuntu-24.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714338/job/63872920720)|
|test-ubuntu-24.04-cpp-bundled-offline|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714069/job/63872919215)|
|test-ubuntu-24.04-cpp-gcc-13-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/22101713735/job/63872918139)|
|test-ubuntu-24.04-cpp-gcc-14|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714688/job/63872921742)|
|test-ubuntu-24.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714049/job/63872919555)|
|test-ubuntu-24.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/22101714717/job/63872922216)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921014991
Thanks for your patience. Conbench analyzed the 3 benchmarking runs that have been run so far on PR commit a3720d4696c4657812f10a8e015e9221f9d88a7c.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64011663606) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3585970849
:warning: GitHub issue #48277 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3588548411
@pitrou I'm running this locally, and I made an error when fixing ASAN over-reading problem.
These benchmarks are not doing well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3922072088
> No, just old compilers (clang 15 and AppleClang 17). For example "Make typename more optional" [appeared in clang 16](https://en.cppreference.com/w/cpp/compiler_support/20.html).
Ok, I can add the `typename` again, but what about the `<concepts>`? Should I revert using them too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793336007
##########
cpp/src/arrow/util/bpacking.cc:
##########
@@ -50,19 +51,19 @@ struct UnpackDynamicFunction {
} // namespace
template <typename Uint>
-void unpack(const uint8_t* in, Uint* out, int batch_size, int num_bits, int bit_offset) {
+void unpack(const uint8_t* in, Uint* out, const UnpackOptions& opts) {
#if defined(ARROW_HAVE_NEON)
- return unpack_neon(in, out, batch_size, num_bits, bit_offset);
+ return bpacking::unpack_neon(in, out, opts);
#else
static DynamicDispatch<UnpackDynamicFunction<Uint> > dispatch;
- return dispatch.func(in, out, batch_size, num_bits, bit_offset);
+ return dispatch.func(in, out, opts);
#endif
}
-template void unpack<bool>(const uint8_t*, bool*, int, int, int);
-template void unpack<uint8_t>(const uint8_t*, uint8_t*, int, int, int);
-template void unpack<uint16_t>(const uint8_t*, uint16_t*, int, int, int);
-template void unpack<uint32_t>(const uint8_t*, uint32_t*, int, int, int);
-template void unpack<uint64_t>(const uint8_t*, uint64_t*, int, int, int);
+template void unpack<bool>(const uint8_t*, bool*, const UnpackOptions&);
Review Comment:
Kind of, though we might want to revisit later anyway. Not necessary for this PR in any case!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793259864
##########
cpp/src/arrow/util/bpacking.cc:
##########
@@ -50,19 +51,19 @@ struct UnpackDynamicFunction {
} // namespace
template <typename Uint>
-void unpack(const uint8_t* in, Uint* out, int batch_size, int num_bits, int bit_offset) {
+void unpack(const uint8_t* in, Uint* out, const UnpackOptions& opts) {
#if defined(ARROW_HAVE_NEON)
- return unpack_neon(in, out, batch_size, num_bits, bit_offset);
+ return bpacking::unpack_neon(in, out, opts);
#else
static DynamicDispatch<UnpackDynamicFunction<Uint> > dispatch;
- return dispatch.func(in, out, batch_size, num_bits, bit_offset);
+ return dispatch.func(in, out, opts);
#endif
}
-template void unpack<bool>(const uint8_t*, bool*, int, int, int);
-template void unpack<uint8_t>(const uint8_t*, uint8_t*, int, int, int);
-template void unpack<uint16_t>(const uint8_t*, uint16_t*, int, int, int);
-template void unpack<uint32_t>(const uint8_t*, uint32_t*, int, int, int);
-template void unpack<uint64_t>(const uint8_t*, uint64_t*, int, int, int);
+template void unpack<bool>(const uint8_t*, bool*, const UnpackOptions&);
Review Comment:
No reason, anything works really. My reasoning was `unpack` is a "library-public" utility function, so it lives in `arrow::internal` while `arrow::internal::bpacking` is "private" to the `unpack` function. Does that makes sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3914902129
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3920902058
There are some errors due to missing `typename` specifiers: https://github.com/ursacomputing/crossbow/actions/runs/22140380450/job/64002821677#step:6:1091
Also some missing concepts here: https://github.com/ursacomputing/crossbow/actions/runs/22140380897/job/64002823560#step:9:1245
@AntoinePrv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921070046
Thanks for your patience. Conbench analyzed the 3 benchmarking runs that have been run so far on PR commit a3720d4696c4657812f10a8e015e9221f9d88a7c.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64012985859) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3589470121
@ursabot please benchmark lang=C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3818916177
Benchmark runs are scheduled for commit a3720d4696c4657812f10a8e015e9221f9d88a7c. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783116838
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
Review Comment:
Yes that is right. I think there are some case where it could be slightly better but not as common as one could imagine.
There is additional complexity to handling it, one is that the current implementation is based on `std::array`s.
Even `constexpr std::vector` would require some adjustments to create `xsimd::batch_constant`, or conversion to `std::array`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2788246388
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ plan.low_swizzles.at(r).at(idx) = packed_byte_in_read + b;
+ plan.high_swizzles.at(r).at(idx) = packed_byte_in_read + b + kOverBytes;
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes created a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+
+ plan.mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+
+ return plan;
+}
+
+template <typename KernelTraits>
+struct LargeKernel {
+ static constexpr auto kPlan = BuildLargePlan<KernelTraits>();
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.kUnpackedPerkernel;
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out) {
+ constexpr auto kLowSwizzles =
+ array_to_batch_constant<kPlan.low_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kLowRShifts =
+ array_to_batch_constant<kPlan.low_rshifts.at(kReadIdx), arch_type>();
+ constexpr auto kHighSwizzles =
+ array_to_batch_constant<kPlan.high_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kHighLShifts =
+ array_to_batch_constant<kPlan.high_lshifts.at(kReadIdx), arch_type>();
+
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+
+ const auto low_swizzled = xsimd::swizzle(bytes, kLowSwizzles);
+ const auto low_words = xsimd::bitwise_cast<unpacked_type>(low_swizzled);
+ simd_batch low_shifted;
+ if constexpr (kShape.unpacked_byte_size() == 1 && HasSse2<arch_type>) {
+ // The logic of the fallback in right_shift_by_excess does not work for this single
+ // byte case case, so we use directly xsimd and its scalar fallback.
+ low_shifted = low_words >> kLowRShifts;
+ } else {
+ low_shifted = right_shift_by_excess(low_words, kLowRShifts);
+ }
+
+ const auto high_swizzled = xsimd::swizzle(bytes, kHighSwizzles);
+ const auto high_words = xsimd::bitwise_cast<unpacked_type>(high_swizzled);
+ const auto high_shifted = left_shift(high_words, kHighLShifts);
+
+ // We can have a single mask and apply it after OR because the shifts will ensure that
+ // there are zeros where the high/low values are incomplete.
+ const auto vals = (low_shifted | high_shifted) & kPlan.mask;
+ xsimd::store_unaligned(out + kReadIdx * kShape.unpacked_per_simd(), vals);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ (unpack_one_read_impl<kReadIds>(in, out), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlan.kReadsPerKernel>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.kUnpackedPerkernel * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*********************
+ * Utility Kernels *
+ *********************/
+
+/// A Kernel that does not extract anything, leaving all work to the naive implementation.
+template <typename KernelTraits>
+struct NoOpKernel {
+ using unpacked_type = typename KernelTraits::unpacked_type;
+
+ static constexpr int kValuesUnpacked = 0;
+ static constexpr int kBytesRead = 0;
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) { return in; }
+};
+
+template <typename KernelTraits, typename WorkingKernel>
+struct ForwardToKernel : WorkingKernel {
+ using unpacked_type = typename KernelTraits::unpacked_type;
+
+ static constexpr int kValuesUnpacked = WorkingKernel::kValuesUnpacked;
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using working_type = typename WorkingKernel::unpacked_type;
+
+ working_type buffer[kValuesUnpacked] = {};
+ in = WorkingKernel::unpack(in, buffer);
+ for (int k = 0; k < kValuesUnpacked; ++k) {
+ out[k] = static_cast<unpacked_type>(buffer[k]);
+ }
+ return in;
+ }
+};
+
+/*******************************
+ * Kernel static dispatching *
+ *******************************/
+
+// Benchmarking show unpack to uint64_t is underperforming on SSE4.2 and Avx2
+template <typename Traits, typename Arch = typename Traits::arch_type>
+constexpr bool MediumShouldUseUint32 =
+ (HasSse2<Arch> || HasSse2<Arch>) && //
Review Comment:
Hmm Ok, I think that is what I meant, but I may not have taken the `static_cast` loop into account here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3937934081
Thanks for your patience. Conbench analyzed the 3 benchmarking runs that have been run so far on PR commit b63857072c2e9da4cf179cde0479150690ee1932.
There were 32 benchmark results indicating a performance regression:
- Pull Request Run on `amd64-m5-4xlarge-linux` at [2026-02-20 15:25:59Z](https://conbench.arrow-dev.org/compare/runs/1a76599f1d25448399bc5fa16bb8644f...a97a8be7f814458f831e5fecfaf7f732/)
- [`BM_UnpackUint64` (C++) with params=Avx2Unaligned/47/64, source=cpp-micro, suite=arrow-bpacking-benchmark](https://conbench.arrow-dev.org/compare/benchmarks/06994b64ec1c799c80000bb870c630b6...069987d954ad70898000e7660c8bb187)
- [`BM_UnpackUint64` (C++) with params=Avx2Unaligned/47/64, source=cpp-micro, suite=arrow-bpacking-benchmark](https://conbench.arrow-dev.org/compare/benchmarks/069988f200c376c7800084146cdece06...069987d954ad70898000e7660c8bb187)
- and 30 more (see the report linked below)
The [full Conbench report](https://github.com/apache/arrow/runs/64366896015) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783198073
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
Review Comment:
Also yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783188395
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
Review Comment:
No, this is the "excess" case that may not produce a shift on all inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2788370595
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
Review Comment:
The require the shape, which is not stored in the PlanSize. Either way, this split is mostly to go play well with compile time computation. The plan size is a regular struct (no template), then instantiated as a `constexpr` var.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3879256205
@pitrou ready again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793549461
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
+/// We therefore cannot end here and reapply the same kernel on the same input.
+/// Instead, we run another similar iteration to unpack E, F, G, H, this time taking into
+/// account that we have an extra offset of 12 bits (this will change all the constants
+/// used for swizzle and shift).
+///
+/// Note that in this example, we could have swizzled more than two values in each slot.
+/// In practice, there may be situations where a more complex algorithm could fit more
+/// shifts per swizzles, but that tend to not be the case when the SIMD register size
+/// increases.
Review Comment:
Yes, in fact the interactive visualizations in the the blog post will help a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783129267
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
Review Comment:
Yes. It usually set to zero, though we do not rely on it here. Bytes that are left undefined are not used (they are eventually masked out).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921058006
Thanks for your patience. Conbench analyzed the 2 benchmarking runs that have been run so far on PR commit b63857072c2e9da4cf179cde0479150690ee1932.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64012731503) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921043723
Thanks for your patience. Conbench analyzed the 3 benchmarking runs that have been run so far on PR commit 97d87e8a5ae6f9d1e337163b20f98a2aa1974c6e.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64012391173) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3602518521
@ursabot please benchmark lang=C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] unpack with shuffle algorithm [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3461837818
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3587245329
Thanks for your patience. Conbench analyzed the 4 benchmarking runs that have been run so far on PR commit a4bfe8addf409c235e0fd96eead5b489447029d0.
There were 37 benchmark results indicating a performance regression:
- Pull Request Run on `amd64-c6a-4xlarge-linux` at [2025-11-27 19:32:37Z](https://conbench.ursa.dev/compare/runs/2d25bf63d48b4d54abb5aa4718492357...f44093f70ebc4d5e83d2d5beee633f81/)
- [`BM_UnpackUint64` (C++) with params=DynamicAligned/47/64, source=cpp-micro, suite=arrow-bpacking-benchmark](https://conbench.ursa.dev/compare/benchmarks/06927c1f3da07b928000f622c3293bcc...06928a8869b774358000a3a785eb5236)
- [`IsInInt64SmallSet` (C++) with params=64, source=cpp-micro, suite=arrow-compute-scalar-set-lookup-benchmark](https://conbench.ursa.dev/compare/benchmarks/06927c17b952761a80005d03b8f1e2d0...06928a81002b73998000e816d8a2211c)
- and 35 more (see the report linked below)
The [full Conbench report](https://github.com/apache/arrow/runs/56584882135) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3818915423
@ursabot please benchmark lang=C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2764820246
##########
cpp/src/arrow/util/bit_stream_utils_internal.h:
##########
@@ -273,14 +273,19 @@ inline int BitReader::GetBatch(int num_bits, T* v, int batch_size) {
batch_size = static_cast<int>(remaining_bits / num_bits);
}
+ const ::arrow::internal::UnpackOptions opts{
+ /* .batch_size= */ batch_size,
+ /* .bit_width= */ num_bits,
+ /* .bit_offset= */ bit_offset_,
+ /* .max_read_bytes= */ max_bytes_ - byte_offset_,
Review Comment:
Now that we use C++20, we can use actual designated initializers :)
##########
cpp/src/arrow/util/bpacking_dispatch_internal.h:
##########
@@ -188,297 +202,431 @@ void unpack_width(const uint8_t* in, UnpackedUInt* out, int batch_size, int bit_
return unpack_full(in, out, batch_size);
} else {
using UnpackerForWidth = Unpacker<UnpackedUInt, kPackedBitWidth>;
+ // Number of values extracted by one iteration of the kernel
constexpr auto kValuesUnpacked = UnpackerForWidth::kValuesUnpacked;
+ // Number of bytes read, but not necessarily unpacked, by one iteration of the
+ // kernel. This constant prevent reading past buffer end.
+ constexpr auto kBytesRead = UnpackerForWidth::kBytesRead;
+
+ if constexpr (kValuesUnpacked > 0) {
+ const uint8_t* in_last = in_end - kBytesRead;
+ // Running the optimized kernel for batch extraction
+ while ((batch_size >= kValuesUnpacked) && (in <= in_last)) {
+ in = UnpackerForWidth::unpack(in, out);
+ out += kValuesUnpacked;
+ batch_size -= kValuesUnpacked;
+ }
- // Running the optimized kernel for batch extraction
- const int unpacker_iter_count = batch_size / kValuesUnpacked;
- for (int i = 0; i < unpacker_iter_count; ++i) {
- in = UnpackerForWidth::unpack(in, out);
- out += kValuesUnpacked;
+ // Performance check aking sure we ran the kernel loop as much as possible:
Review Comment:
```suggestion
// Performance check making sure we ran the kernel loop as much as possible:
```
##########
cpp/src/arrow/util/bpacking_dispatch_internal.h:
##########
@@ -166,13 +167,26 @@ int unpack_exact(const uint8_t* in, Uint* out, int batch_size, int bit_offset) {
/// fixed amount of values (usually constrained by SIMD batch sizes and
/// byte alignment).
/// @tparam UnpackedUInt The type in which we unpack the values.
+/// @param batch_size The number of values to unpack.
+/// @param max_read_bytes The maximum size of the input byte array that can be read.
+/// This is used to safely overread.
+/// Negative value to deduce from batch_size.
template <int kPackedBitWidth, template <typename, int> typename Unpacker,
typename UnpackedUInt>
-void unpack_width(const uint8_t* in, UnpackedUInt* out, int batch_size, int bit_offset) {
+void unpack_width(const uint8_t* in, UnpackedUInt* out, int batch_size, int bit_offset,
+ int max_read_bytes) {
if constexpr (kPackedBitWidth == 0) {
// Easy case to handle, simply setting memory to zero.
return unpack_null(in, out, batch_size);
} else {
+ // Number of size to read according to batch_size.
Review Comment:
Do you mean "number of bytes"?
##########
cpp/src/arrow/util/bpacking_benchmark.cc:
##########
@@ -86,33 +87,62 @@ void BM_Unpack(benchmark::State& state, bool aligned, UnpackFunc<Int> unpack, bo
const uint8_t* packed_ptr =
GetNextAlignedByte(packed.data(), sizeof(Int)) + (aligned ? 0 : 1);
- std::vector<Int> unpacked(num_values, 0);
+ auto unpacked = std::make_unique<Int[]>(num_values);
+
+ const ::arrow::internal::UnpackOptions opts{
+ /* .batch_size= */ num_values,
+ /* .bit_width= */ bit_width,
+ /* .bit_offset= */ 0,
+ /* .max_read_bytes= */ -1,
+ };
for (auto _ : state) {
- unpack(packed_ptr, unpacked.data(), num_values, bit_width, /* bit_offset = */ 0);
+ unpack(packed_ptr, unpacked.get(), opts);
benchmark::ClobberMemory();
}
state.SetItemsProcessed(num_values * state.iterations());
}
-constexpr int32_t kMinRange = 64;
-constexpr int32_t kMaxRange = 32768;
+/// Currently, the minimum unpack SIMD kernel size is 32 and the bit packing encoder will
Review Comment:
```suggestion
/// Currently, the minimum unpack SIMD kernel size is 32 and the RLE-bit-packing encoder will
```
##########
cpp/src/arrow/util/bpacking_dispatch_internal.h:
##########
@@ -166,13 +167,26 @@ int unpack_exact(const uint8_t* in, Uint* out, int batch_size, int bit_offset) {
/// fixed amount of values (usually constrained by SIMD batch sizes and
/// byte alignment).
/// @tparam UnpackedUInt The type in which we unpack the values.
+/// @param batch_size The number of values to unpack.
Review Comment:
If you document any function arguments, you'll probably have to document all of them, or some compilers will error out.
##########
cpp/src/arrow/util/bpacking_simd_default.cc:
##########
@@ -15,28 +15,34 @@
// specific language governing permissions and limitations
// under the License.
-#include "arrow/util/bpacking_dispatch_internal.h"
#if defined(ARROW_HAVE_NEON)
-# include "arrow/util/bpacking_simd128_generated_internal.h"
+# define UNPACK_PLATFORM unpack_neon
+#elif defined(ARROW_HAVE_SSE4_2)
+# define UNPACK_PLATFORM unpack_sse4_2
#endif
-#include "arrow/util/bpacking_simd_internal.h"
+
+#if defined(UNPACK_PLATFORM)
+
+# include "arrow/util/bpacking_dispatch_internal.h"
+# include "arrow/util/bpacking_simd_internal.h"
+# include "arrow/util/bpacking_simd_kernel_internal.h"
namespace arrow::internal {
-#if defined(ARROW_HAVE_NEON)
+template <typename UnpackedUint, int kPackedBitSize>
+using Simd128Kernel = Kernel<UnpackedUint, kPackedBitSize, 128>;
template <typename Uint>
-void unpack_neon(const uint8_t* in, Uint* out, int batch_size, int num_bits,
- int bit_offset) {
- return unpack_jump<Simd128UnpackerForWidth>(in, out, batch_size, num_bits, bit_offset);
+void UNPACK_PLATFORM(const uint8_t* in, Uint* out, const UnpackOptions& opts) {
+ return unpack_jump<Simd128Kernel>(in, out, opts);
}
-template void unpack_neon<bool>(const uint8_t*, bool*, int, int, int);
-template void unpack_neon<uint8_t>(const uint8_t*, uint8_t*, int, int, int);
-template void unpack_neon<uint16_t>(const uint8_t*, uint16_t*, int, int, int);
-template void unpack_neon<uint32_t>(const uint8_t*, uint32_t*, int, int, int);
-template void unpack_neon<uint64_t>(const uint8_t*, uint64_t*, int, int, int);
-
-#endif
+template void UNPACK_PLATFORM<bool>(const uint8_t*, bool*, const UnpackOptions&);
+template void UNPACK_PLATFORM<uint8_t>(const uint8_t*, uint8_t*, const UnpackOptions&);
+template void UNPACK_PLATFORM<uint16_t>(const uint8_t*, uint16_t*, const UnpackOptions&);
+template void UNPACK_PLATFORM<uint32_t>(const uint8_t*, uint32_t*, const UnpackOptions&);
+template void UNPACK_PLATFORM<uint64_t>(const uint8_t*, uint64_t*, const UnpackOptions&);
Review Comment:
Can you undefine `UNPACK_PLATFORM` at the end? Otherwise it can leak into other files in unity builds.
##########
cpp/src/arrow/util/bpacking_benchmark.cc:
##########
@@ -86,33 +87,62 @@ void BM_Unpack(benchmark::State& state, bool aligned, UnpackFunc<Int> unpack, bo
const uint8_t* packed_ptr =
GetNextAlignedByte(packed.data(), sizeof(Int)) + (aligned ? 0 : 1);
- std::vector<Int> unpacked(num_values, 0);
+ auto unpacked = std::make_unique<Int[]>(num_values);
+
+ const ::arrow::internal::UnpackOptions opts{
+ /* .batch_size= */ num_values,
+ /* .bit_width= */ bit_width,
+ /* .bit_offset= */ 0,
+ /* .max_read_bytes= */ -1,
+ };
for (auto _ : state) {
- unpack(packed_ptr, unpacked.data(), num_values, bit_width, /* bit_offset = */ 0);
+ unpack(packed_ptr, unpacked.get(), opts);
benchmark::ClobberMemory();
}
state.SetItemsProcessed(num_values * state.iterations());
}
-constexpr int32_t kMinRange = 64;
-constexpr int32_t kMaxRange = 32768;
+/// Currently, the minimum unpack SIMD kernel size is 32 and the bit packing encoder will
+/// not emit runs larger than 512 (though other implementation might), so we biased the
Review Comment:
Nit: triple slashes are for docstrings, not plain comments :)
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
Review Comment:
Since there's a ton of specific utilities, can we use a more a specific namespace too? Something like `arrow::internal::bpacking` or `arrow::bpacking::internal`, at your choice :)
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
Review Comment:
I see this xsimd PR is in limbo, should it be revived?
(but we can of course have this code here in the meantime)
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
Review Comment:
Should we `static_assert` that `simd_bit_size` is a multiple of `unpacked_bit_size`?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
Review Comment:
```suggestion
/// When integers are bit-packed, they can spread over multiple bytes.
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
Review Comment:
This is not easy to understand. One can certainly fit at least 63 bits on 128 bit SIMD?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
Review Comment:
```suggestion
// A single SIMD batch isn't enough to unpack from this bit offset,
// so the "medium" model cannot work.
return true;
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
Review Comment:
How about making this a `static` factory for better readability, e.g. `MediumKernelPlanSize::Build`?
##########
cpp/src/arrow/util/bpacking_dispatch_internal.h:
##########
@@ -188,297 +202,431 @@ void unpack_width(const uint8_t* in, UnpackedUInt* out, int batch_size, int bit_
return unpack_full(in, out, batch_size);
} else {
using UnpackerForWidth = Unpacker<UnpackedUInt, kPackedBitWidth>;
+ // Number of values extracted by one iteration of the kernel
constexpr auto kValuesUnpacked = UnpackerForWidth::kValuesUnpacked;
+ // Number of bytes read, but not necessarily unpacked, by one iteration of the
+ // kernel. This constant prevent reading past buffer end.
+ constexpr auto kBytesRead = UnpackerForWidth::kBytesRead;
+
+ if constexpr (kValuesUnpacked > 0) {
+ const uint8_t* in_last = in_end - kBytesRead;
+ // Running the optimized kernel for batch extraction
+ while ((batch_size >= kValuesUnpacked) && (in <= in_last)) {
Review Comment:
Not necessary for this PR, but I wonder if making the `while` loop part of the `UnpackerForWidth::unpack` implementation could improve performance slightly. Just something to think about perhaps.
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
Review Comment:
```suggestion
// Try all possible bit offsets for this packed size
do {
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
Review Comment:
It would very useful to add comments explaining, even very briefly, how a "medium" kernel works.
Also if it's possible, a fictional example with small sizes to make it readable (for example: unpack 3-bit ints into 8-bit ints using 16-bit SIMD registers).
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
Review Comment:
Let's add an actual description?
```suggestion
/// Compute whether the "medium" model would work for these bit sizes
/// \see KernelShape
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
Review Comment:
So we're going to do the same number of swizzles for each bit offset, instead of a different number for each? Is that right? Are we leaving performance on the table?
(I might misunderstand how this works)
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
Review Comment:
You mean byte-unaligned, not SIMD-unaligned, right?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
Review Comment:
Make this a static factory method too?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
Review Comment:
```suggestion
static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
Review Comment:
How about simply calling it `right_shift`? For the caller, it doesn't make any difference how the right shift is actually implemented, right?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
Review Comment:
Is there also a xsimd PR for this? If so, mention it?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
Review Comment:
1. Can we move this to the general utilities section at the start of the file?
2. The templating is for it to work on both std::array and SIMD batches?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
Review Comment:
Should these informative methods be part of `MediumKernelPlanSize` instead? It seems we don't need to instantiate an actual kernel to compute them.
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
Review Comment:
I'm afraid I don't understand this comment.
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
Review Comment:
So no `LargeKernelPlanSize`? I think it would be more readable to have the same separation of concerns as for the medium plan.
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
Review Comment:
Should we assert that `kUnpackedPerkernel` is a multiple of `kReadsPerKernel`?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ plan.low_swizzles.at(r).at(idx) = packed_byte_in_read + b;
+ plan.high_swizzles.at(r).at(idx) = packed_byte_in_read + b + kOverBytes;
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes created a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+
+ plan.mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+
+ return plan;
+}
+
+template <typename KernelTraits>
+struct LargeKernel {
+ static constexpr auto kPlan = BuildLargePlan<KernelTraits>();
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.kUnpackedPerkernel;
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out) {
+ constexpr auto kLowSwizzles =
+ array_to_batch_constant<kPlan.low_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kLowRShifts =
+ array_to_batch_constant<kPlan.low_rshifts.at(kReadIdx), arch_type>();
+ constexpr auto kHighSwizzles =
+ array_to_batch_constant<kPlan.high_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kHighLShifts =
+ array_to_batch_constant<kPlan.high_lshifts.at(kReadIdx), arch_type>();
+
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+
+ const auto low_swizzled = xsimd::swizzle(bytes, kLowSwizzles);
+ const auto low_words = xsimd::bitwise_cast<unpacked_type>(low_swizzled);
+ simd_batch low_shifted;
+ if constexpr (kShape.unpacked_byte_size() == 1 && HasSse2<arch_type>) {
+ // The logic of the fallback in right_shift_by_excess does not work for this single
+ // byte case case, so we use directly xsimd and its scalar fallback.
+ low_shifted = low_words >> kLowRShifts;
+ } else {
+ low_shifted = right_shift_by_excess(low_words, kLowRShifts);
+ }
+
+ const auto high_swizzled = xsimd::swizzle(bytes, kHighSwizzles);
+ const auto high_words = xsimd::bitwise_cast<unpacked_type>(high_swizzled);
+ const auto high_shifted = left_shift(high_words, kHighLShifts);
+
+ // We can have a single mask and apply it after OR because the shifts will ensure that
+ // there are zeros where the high/low values are incomplete.
+ const auto vals = (low_shifted | high_shifted) & kPlan.mask;
+ xsimd::store_unaligned(out + kReadIdx * kShape.unpacked_per_simd(), vals);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ (unpack_one_read_impl<kReadIds>(in, out), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlan.kReadsPerKernel>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.kUnpackedPerkernel * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*********************
+ * Utility Kernels *
+ *********************/
+
+/// A Kernel that does not extract anything, leaving all work to the naive implementation.
+template <typename KernelTraits>
+struct NoOpKernel {
+ using unpacked_type = typename KernelTraits::unpacked_type;
+
+ static constexpr int kValuesUnpacked = 0;
+ static constexpr int kBytesRead = 0;
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) { return in; }
+};
+
+template <typename KernelTraits, typename WorkingKernel>
+struct ForwardToKernel : WorkingKernel {
+ using unpacked_type = typename KernelTraits::unpacked_type;
+
+ static constexpr int kValuesUnpacked = WorkingKernel::kValuesUnpacked;
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using working_type = typename WorkingKernel::unpacked_type;
+
+ working_type buffer[kValuesUnpacked] = {};
+ in = WorkingKernel::unpack(in, buffer);
+ for (int k = 0; k < kValuesUnpacked; ++k) {
+ out[k] = static_cast<unpacked_type>(buffer[k]);
+ }
+ return in;
+ }
+};
+
+/*******************************
+ * Kernel static dispatching *
+ *******************************/
+
+// Benchmarking show unpack to uint64_t is underperforming on SSE4.2 and Avx2
+template <typename Traits, typename Arch = typename Traits::arch_type>
+constexpr bool MediumShouldUseUint32 =
+ (HasSse2<Arch> || HasSse2<Arch>) && //
Review Comment:
Well, I've tried `HasAvx2` here and I see only regressions on my AMD Zen 2 CPU (up to -40% on some bit widths). So we should perhaps not add it :)
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
Review Comment:
Which hierarchy? Can you make the message a bit more explicit?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
Review Comment:
Make it a static factory function?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ plan.low_swizzles.at(r).at(idx) = packed_byte_in_read + b;
+ plan.high_swizzles.at(r).at(idx) = packed_byte_in_read + b + kOverBytes;
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes created a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+
+ plan.mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+
+ return plan;
+}
+
+template <typename KernelTraits>
+struct LargeKernel {
+ static constexpr auto kPlan = BuildLargePlan<KernelTraits>();
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.kUnpackedPerkernel;
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out) {
+ constexpr auto kLowSwizzles =
+ array_to_batch_constant<kPlan.low_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kLowRShifts =
+ array_to_batch_constant<kPlan.low_rshifts.at(kReadIdx), arch_type>();
+ constexpr auto kHighSwizzles =
+ array_to_batch_constant<kPlan.high_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kHighLShifts =
+ array_to_batch_constant<kPlan.high_lshifts.at(kReadIdx), arch_type>();
+
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+
+ const auto low_swizzled = xsimd::swizzle(bytes, kLowSwizzles);
+ const auto low_words = xsimd::bitwise_cast<unpacked_type>(low_swizzled);
+ simd_batch low_shifted;
+ if constexpr (kShape.unpacked_byte_size() == 1 && HasSse2<arch_type>) {
+ // The logic of the fallback in right_shift_by_excess does not work for this single
+ // byte case case, so we use directly xsimd and its scalar fallback.
+ low_shifted = low_words >> kLowRShifts;
+ } else {
+ low_shifted = right_shift_by_excess(low_words, kLowRShifts);
+ }
+
+ const auto high_swizzled = xsimd::swizzle(bytes, kHighSwizzles);
+ const auto high_words = xsimd::bitwise_cast<unpacked_type>(high_swizzled);
+ const auto high_shifted = left_shift(high_words, kHighLShifts);
+
+ // We can have a single mask and apply it after OR because the shifts will ensure that
+ // there are zeros where the high/low values are incomplete.
+ const auto vals = (low_shifted | high_shifted) & kPlan.mask;
+ xsimd::store_unaligned(out + kReadIdx * kShape.unpacked_per_simd(), vals);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ (unpack_one_read_impl<kReadIds>(in, out), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlan.kReadsPerKernel>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.kUnpackedPerkernel * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*********************
+ * Utility Kernels *
+ *********************/
+
+/// A Kernel that does not extract anything, leaving all work to the naive implementation.
+template <typename KernelTraits>
+struct NoOpKernel {
+ using unpacked_type = typename KernelTraits::unpacked_type;
+
+ static constexpr int kValuesUnpacked = 0;
+ static constexpr int kBytesRead = 0;
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) { return in; }
+};
+
+template <typename KernelTraits, typename WorkingKernel>
+struct ForwardToKernel : WorkingKernel {
Review Comment:
Perhaps call it `CastingKernel`, something like:
```c++
template <typename WorkingKernel, typename UnpackUint>
struct CastingKernel : WorkingKernel {
using unpacked_type = UnpackUint;
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
Review Comment:
Can then reduce performance on the platforms that do have variable right shifts, or is that not a concern?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
Review Comment:
Add a docstring here?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
Review Comment:
```suggestion
/// For instance, integers packed over three bits quickly spread over two bytes (on the
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
Review Comment:
Is there a "small" model? If so, describe it too?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
Review Comment:
This will be the unpacked int type, right?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
Review Comment:
Hmm, so `Traits` is assumed to be a `KernelTraits` instance already?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ plan.low_swizzles.at(r).at(idx) = packed_byte_in_read + b;
+ plan.high_swizzles.at(r).at(idx) = packed_byte_in_read + b + kOverBytes;
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes created a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+
+ plan.mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+
+ return plan;
+}
+
+template <typename KernelTraits>
+struct LargeKernel {
+ static constexpr auto kPlan = BuildLargePlan<KernelTraits>();
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.kUnpackedPerkernel;
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out) {
+ constexpr auto kLowSwizzles =
+ array_to_batch_constant<kPlan.low_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kLowRShifts =
+ array_to_batch_constant<kPlan.low_rshifts.at(kReadIdx), arch_type>();
+ constexpr auto kHighSwizzles =
+ array_to_batch_constant<kPlan.high_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kHighLShifts =
+ array_to_batch_constant<kPlan.high_lshifts.at(kReadIdx), arch_type>();
+
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+
+ const auto low_swizzled = xsimd::swizzle(bytes, kLowSwizzles);
+ const auto low_words = xsimd::bitwise_cast<unpacked_type>(low_swizzled);
+ simd_batch low_shifted;
+ if constexpr (kShape.unpacked_byte_size() == 1 && HasSse2<arch_type>) {
+ // The logic of the fallback in right_shift_by_excess does not work for this single
+ // byte case case, so we use directly xsimd and its scalar fallback.
+ low_shifted = low_words >> kLowRShifts;
+ } else {
+ low_shifted = right_shift_by_excess(low_words, kLowRShifts);
+ }
+
+ const auto high_swizzled = xsimd::swizzle(bytes, kHighSwizzles);
+ const auto high_words = xsimd::bitwise_cast<unpacked_type>(high_swizzled);
+ const auto high_shifted = left_shift(high_words, kHighLShifts);
+
+ // We can have a single mask and apply it after OR because the shifts will ensure that
+ // there are zeros where the high/low values are incomplete.
+ const auto vals = (low_shifted | high_shifted) & kPlan.mask;
+ xsimd::store_unaligned(out + kReadIdx * kShape.unpacked_per_simd(), vals);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ (unpack_one_read_impl<kReadIds>(in, out), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlan.kReadsPerKernel>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.kUnpackedPerkernel * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*********************
+ * Utility Kernels *
+ *********************/
+
+/// A Kernel that does not extract anything, leaving all work to the naive implementation.
+template <typename KernelTraits>
+struct NoOpKernel {
+ using unpacked_type = typename KernelTraits::unpacked_type;
+
+ static constexpr int kValuesUnpacked = 0;
+ static constexpr int kBytesRead = 0;
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) { return in; }
+};
+
+template <typename KernelTraits, typename WorkingKernel>
+struct ForwardToKernel : WorkingKernel {
+ using unpacked_type = typename KernelTraits::unpacked_type;
+
+ static constexpr int kValuesUnpacked = WorkingKernel::kValuesUnpacked;
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using working_type = typename WorkingKernel::unpacked_type;
+
+ working_type buffer[kValuesUnpacked] = {};
+ in = WorkingKernel::unpack(in, buffer);
+ for (int k = 0; k < kValuesUnpacked; ++k) {
+ out[k] = static_cast<unpacked_type>(buffer[k]);
+ }
+ return in;
+ }
+};
+
+/*******************************
+ * Kernel static dispatching *
+ *******************************/
+
+// Benchmarking show unpack to uint64_t is underperforming on SSE4.2 and Avx2
+template <typename Traits, typename Arch = typename Traits::arch_type>
+constexpr bool MediumShouldUseUint32 =
Review Comment:
Let's use our naming convention for constants, e.g. `kMediumShouldUseUint32`.
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ plan.low_swizzles.at(r).at(idx) = packed_byte_in_read + b;
+ plan.high_swizzles.at(r).at(idx) = packed_byte_in_read + b + kOverBytes;
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes created a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
Review Comment:
Should we assert that packed_start_bit % 8 is 0 here?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ plan.low_swizzles.at(r).at(idx) = packed_byte_in_read + b;
+ plan.high_swizzles.at(r).at(idx) = packed_byte_in_read + b + kOverBytes;
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes created a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+
+ plan.mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+
+ return plan;
+}
+
+template <typename KernelTraits>
+struct LargeKernel {
+ static constexpr auto kPlan = BuildLargePlan<KernelTraits>();
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.kUnpackedPerkernel;
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out) {
+ constexpr auto kLowSwizzles =
+ array_to_batch_constant<kPlan.low_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kLowRShifts =
+ array_to_batch_constant<kPlan.low_rshifts.at(kReadIdx), arch_type>();
+ constexpr auto kHighSwizzles =
+ array_to_batch_constant<kPlan.high_swizzles.at(kReadIdx), arch_type>();
+ constexpr auto kHighLShifts =
+ array_to_batch_constant<kPlan.high_lshifts.at(kReadIdx), arch_type>();
+
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+
+ const auto low_swizzled = xsimd::swizzle(bytes, kLowSwizzles);
+ const auto low_words = xsimd::bitwise_cast<unpacked_type>(low_swizzled);
+ simd_batch low_shifted;
+ if constexpr (kShape.unpacked_byte_size() == 1 && HasSse2<arch_type>) {
+ // The logic of the fallback in right_shift_by_excess does not work for this single
+ // byte case case, so we use directly xsimd and its scalar fallback.
+ low_shifted = low_words >> kLowRShifts;
+ } else {
+ low_shifted = right_shift_by_excess(low_words, kLowRShifts);
+ }
+
+ const auto high_swizzled = xsimd::swizzle(bytes, kHighSwizzles);
+ const auto high_words = xsimd::bitwise_cast<unpacked_type>(high_swizzled);
+ const auto high_shifted = left_shift(high_words, kHighLShifts);
+
+ // We can have a single mask and apply it after OR because the shifts will ensure that
+ // there are zeros where the high/low values are incomplete.
+ const auto vals = (low_shifted | high_shifted) & kPlan.mask;
+ xsimd::store_unaligned(out + kReadIdx * kShape.unpacked_per_simd(), vals);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ (unpack_one_read_impl<kReadIds>(in, out), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlan.kReadsPerKernel>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.kUnpackedPerkernel * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*********************
+ * Utility Kernels *
+ *********************/
+
+/// A Kernel that does not extract anything, leaving all work to the naive implementation.
+template <typename KernelTraits>
+struct NoOpKernel {
+ using unpacked_type = typename KernelTraits::unpacked_type;
+
+ static constexpr int kValuesUnpacked = 0;
+ static constexpr int kBytesRead = 0;
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) { return in; }
+};
+
+template <typename KernelTraits, typename WorkingKernel>
+struct ForwardToKernel : WorkingKernel {
+ using unpacked_type = typename KernelTraits::unpacked_type;
+
+ static constexpr int kValuesUnpacked = WorkingKernel::kValuesUnpacked;
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using working_type = typename WorkingKernel::unpacked_type;
+
+ working_type buffer[kValuesUnpacked] = {};
+ in = WorkingKernel::unpack(in, buffer);
+ for (int k = 0; k < kValuesUnpacked; ++k) {
+ out[k] = static_cast<unpacked_type>(buffer[k]);
+ }
+ return in;
+ }
+};
+
+/*******************************
+ * Kernel static dispatching *
+ *******************************/
+
+// Benchmarking show unpack to uint64_t is underperforming on SSE4.2 and Avx2
+template <typename Traits, typename Arch = typename Traits::arch_type>
+constexpr bool MediumShouldUseUint32 =
+ (HasSse2<Arch> || HasSse2<Arch>) && //
Review Comment:
Bug here? Do you mean `HasSse2<Arch> || HasAvx2<Arch>`?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
Review Comment:
Can this overwrite an already defined value? Can we assert that we are not doing anything wrong?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
Review Comment:
What about this one? Byte offsets too?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
Review Comment:
Will some swizzle entries be left undefined? If so, what does it mean? And if not, can we assert that they are all defined at the end?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
Review Comment:
Explain this function?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
Review Comment:
Can we call is `IsSse2` instead, because intuitively AVX2 implies SSE2, but that will be false here?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
Review Comment:
What are these? Byte offsets? Please comment.
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
Review Comment:
Probably also `IsAvx2`, because AVX-512 will make this false.
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
Review Comment:
Move these utilities to the general utilities section at the top?
(we may even want to have a `arrow/util/simd_internal.h` for this kind of stuff)
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
Review Comment:
Should `packed_start_bit` be 0 at the end? Can we assert that and other relevant postconditions?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
Review Comment:
Same question as for the medium kernel: can some swizzle values be left undefined? If not, can we assert that?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
Review Comment:
Similarly as the medium kernel, I would like some brief explanation, and perhaps a small fictional example (such as 7-bit packed values to 8-bit unpacked values with 16-bit SIMD?).
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ plan.low_swizzles.at(r).at(idx) = packed_byte_in_read + b;
+ plan.high_swizzles.at(r).at(idx) = packed_byte_in_read + b + kOverBytes;
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes created a
+ // larger lshift for high values.
Review Comment:
```suggestion
// low and high swizzles need to be rshifted but the oversized bytes create a
// larger lshift for high values.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793338183
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
Review Comment:
This is a limitation of the current algorithm. When we swizzle, we assume the worst byte spread (here 2) for all values. It does not care what is the in the other swizzled bytes, only adds their own. This is why we are doing `sizeof(uint32_t) / max_spread = 4 / 2 = 2` shifts per swizzle and not 5 (from your next comment).
So below are all three steps one next to the other that determine what goes where
```
Val we want. | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
Byte idx for val: | Idx 0 | Idx 1 | Idx 0 | Idx 1 || Idx 0 | Idx 1 | Idx 1 | Idx 2 |
Result data: |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
```
I'll add a comment to clarify that.
See next comment for discussion the improvements to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2793518658
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
Review Comment:
First see my previous comment to see how the current algorithm is implemented.
About the improvement: yes, it's possible to try to pack more "shifts" in a "swizzle" by also considering what previous indices are.
However this is a more complex algorithm to implement, as we have to guarantee that each "slot" has the same number of values. It is also much more dynamic. When this was done Arrow was in C++17 with no `constexpr std::vector`.
Finally, it may seem on this example that we are leaving a lot on the table but that is not the case in practice.
The extremely small "SIMD" register in the example makes it much more likely that we could reuse previous bytes. But in practice, with more "slots" in the register, it becomes unlikely that the next value that must go in a given slot will share bytes with the previous one, and more likely that some value get plagued by a "max spread" case.
All this will go with the "small kernel" ideas not implemented. I believe we are talking gaining one more shift per swizzle. Perhaps more an extreme cases such as `uint64_t` on SIMD128 (currently using `uint32_t` for small size s at the moment in fact).
There are also advanced improvements to `xsimd::swizzle` that we can do to shave one instruction. All these have diminishing returns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783043956
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
Review Comment:
This is the working integer type, most of the type it is equal to `unpacked_type`, but is an integer of equivalent size for `bool`. See doc in `KernelTraits`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783595315
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
Review Comment:
We don't make a difference between SSE2 and SSE4.2 in Arrow (it's always the latter). The important distinction here is between "SSE4.2 only" and "AVX2".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3916288205
@ursabot please benchmark lang=C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921050758
Thanks for your patience. Conbench analyzed the 3 benchmarking runs that have been run so far on PR commit 97d87e8a5ae6f9d1e337163b20f98a2aa1974c6e.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64012569835) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3920971875
Could these build still be compiling with C++17?
> There are some errors due to missing `typename` specifiers: https://github.com/ursacomputing/crossbow/actions/runs/22140380450/job/64002821677#step:6:1091
I explicitly removed them because they are not needed in C++20. It is more readable but I can add them again.
> Also some missing concepts here: https://github.com/ursacomputing/crossbow/actions/runs/22140380897/job/64002823560#step:9:1245
The header seem to be properly included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921093659
Thanks for your patience. Conbench analyzed the 3 benchmarking runs that have been run so far on PR commit 97d87e8a5ae6f9d1e337163b20f98a2aa1974c6e.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64013485519) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3920664258
Revision: b63857072c2e9da4cf179cde0479150690ee1932
Submitted crossbow builds: [ursacomputing/crossbow @ actions-5da0772e05](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5da0772e05)
|Task|Status|
|----|------|
|r-binary-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/22140381490/job/64002903600)|
|r-recheck-most|[](https://github.com/ursacomputing/crossbow/actions/runs/22140381178/job/64002824448)|
|test-r-alpine-linux-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/22140381394/job/64002825205)|
|test-r-arrow-backwards-compatibility|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380559/job/64002822147)|
|test-r-depsource-system|[](https://github.com/ursacomputing/crossbow/actions/runs/22140381216/job/64002824513)|
|test-r-dev-duckdb|[](https://github.com/ursacomputing/crossbow/actions/runs/22140381067/job/64002824086)|
|test-r-devdocs|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380255/job/64002821242)|
|test-r-extra-packages|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380765/job/64002822920)|
|test-r-fedora-clang|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380412/job/64002821659)|
|test-r-gcc-11|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380952/job/64002823910)|
|test-r-gcc-12|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380887/job/64002823706)|
|test-r-install-local|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380309/job/64002821439)|
|test-r-install-local-minsizerel|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380382/job/64002821679)|
|test-r-linux-as-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380450/job/64002821852)|
|test-r-linux-rchk|[](https://github.com/ursacomputing/crossbow/actions/runs/22140381449/job/64002825418)|
|test-r-linux-sanitizers|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380648/job/64002822519)|
|test-r-linux-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380486/job/64002822015)|
|test-r-m1-san|[](https://github.com/ursacomputing/crossbow/actions/runs/22140381470/job/64002825630)|
|test-r-macos-as-cran|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380897/job/64002823597)|
|test-r-offline-maximal|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380251/job/64002821249)|
|test-r-ubuntu-22.04|[](https://github.com/ursacomputing/crossbow/actions/runs/22140380762/job/64002822899)|
|test-r-versions|[](https://github.com/ursacomputing/crossbow/actions/runs/22140381469/job/64002825720)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3649551100
Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 408ef04ad96a9752654cfe54d4de6c7c2eef08cc.
None of the specified runs were found on the Conbench server.
The [full Conbench report](https://github.com/apache/arrow/runs/57975884615) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3915857979
@jonkeane How would you go about diagnosing a R test that crashes/stops with no error message on Windows (see [above](https://github.com/apache/arrow/pull/47994#issuecomment-3915841768))?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3916289031
Benchmark runs are scheduled for commit b63857072c2e9da4cf179cde0479150690ee1932. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "voltrondatabot (via GitHub)" <gi...@apache.org>.
voltrondatabot commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3586990600
Benchmark runs are scheduled for commit a4bfe8addf409c235e0fd96eead5b489447029d0. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3665536858
@ursabot please benchmark lang=C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3638599502
Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 408ef04ad96a9752654cfe54d4de6c7c2eef08cc.
None of the specified runs were found on the Conbench server.
The [full Conbench report](https://github.com/apache/arrow/runs/57706186944) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2768506186
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
Review Comment:
Ah, scratch that, it doesn't work in all cases, right? Is it possible to somehow detect an invalid case and assert out of it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783197094
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kHasAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kHasSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// \see KernelShape
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they spread over multiple bytes.
+/// For instance, integers packed voer three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium". When it is strictly larger it classifies as "large".
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch (mainly unpack 63
+/// bits to uint64_t on 128 bit SIMD). We classify this as "oversized".
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <typename UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ using unpacked_type = UnpackedUint;
+ // The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = typename simd_batch::arch_type;
+};
+
+template <typename Traits, typename Uint>
+using KernelTraitsWithUnpack =
+ KernelTraits<Uint, Traits::kShape.packed_bit_size(), Traits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+constexpr MediumKernelPlanSize BuildMediumPlanSize(const KernelShape& shape,
+ const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall unaligned, the remaining
+ // iterations will start with an unaligned first value, reducing the effective capacity
+ // of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned.
+ // One may think that using such large reading iterations risks overshooting an aligned
+ // byte and increasing the total number of values extracted, but in practice reading
+ // iterations increase by factors of 2 and are quickly multiple of 8 (and aligned after
+ // the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+constexpr int reduced_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = BuildMediumPlanSize(kShape, kOptions);
+
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shift, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShitsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ static constexpr int unpacked_per_shifts() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shifts() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShitsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename Arr>
+constexpr Arr BuildConstantArray(typename Arr::value_type val) {
+ Arr out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+constexpr auto BuildMediumPlan() {
+ using Plan = MediumKernelPlan<KernelTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ constexpr int kUndefined = -1;
+ plan.swizzles.at(r).at(sw) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+
+ // Looping over the multiple bytes needed for current values
+ for (int b = 0; b < kShape.packed_max_spread_bytes(); ++b) {
+ plan.swizzles.at(r).at(sw).at(sw_offset_byte + b) =
+ static_cast<uint8_t>(packed_byte_in_read + b);
+ }
+ // Shift is a single value but many packed values may be swizzles to a sing
+ // unpacked value
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ return plan;
+}
+
+template <typename Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_bytes_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <typename KernelTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = BuildMediumPlan<KernelTraits, kOptions>();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = typename decltype(kPlan)::Traits;
+ using unpacked_type = typename Traits::unpacked_type;
+ using uint_type = typename Traits::uint_type;
+ using simd_batch = typename Traits::simd_batch;
+ using simd_bytes = typename Traits::simd_bytes;
+ using arch_type = typename Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shifts());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+template <typename KernelTraits>
+struct LargeKernelPlan {
+ using Traits = KernelTraits;
+ using uint_type = typename Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+
+ static constexpr int kUnpackedPerkernel = std::lcm(kShape.unpacked_per_simd(), 8);
+ static constexpr int kReadsPerKernel = static_cast<int>(bit_util::CeilDiv(
+ kUnpackedPerkernel * kShape.packed_bit_size(), kShape.simd_bit_size()));
+ static constexpr int kUnpackedPerRead = kUnpackedPerkernel / kReadsPerKernel;
+
+ using ReadsPerKernel = std::array<int, kReadsPerKernel>;
+
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kReadsPerKernel>;
+
+ using Shift = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shift, kReadsPerKernel>;
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kUnpackedPerRead * kShape.packed_bit_size();
+ return reduced_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KernelTraits>
+constexpr LargeKernelPlan<KernelTraits> BuildLargePlan() {
+ using Plan = LargeKernelPlan<KernelTraits>;
+ using uint_type = typename Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < Plan::kReadsPerKernel; ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ constexpr int kUndefined = -1;
+ plan.low_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArray<typename Plan::Swizzle>(kUndefined);
+
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_byte_in_read = packed_start_byte - read_start_byte;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
Review Comment:
Not really, this just means that some part of the values to unpack will be in the high and low part.
But anyways we have to put something in the swizzle, so in the end it does not really matter what goes in there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2788999901
##########
cpp/src/arrow/util/bpacking.cc:
##########
@@ -50,19 +51,19 @@ struct UnpackDynamicFunction {
} // namespace
template <typename Uint>
-void unpack(const uint8_t* in, Uint* out, int batch_size, int num_bits, int bit_offset) {
+void unpack(const uint8_t* in, Uint* out, const UnpackOptions& opts) {
#if defined(ARROW_HAVE_NEON)
- return unpack_neon(in, out, batch_size, num_bits, bit_offset);
+ return bpacking::unpack_neon(in, out, opts);
#else
static DynamicDispatch<UnpackDynamicFunction<Uint> > dispatch;
- return dispatch.func(in, out, batch_size, num_bits, bit_offset);
+ return dispatch.func(in, out, opts);
#endif
}
-template void unpack<bool>(const uint8_t*, bool*, int, int, int);
-template void unpack<uint8_t>(const uint8_t*, uint8_t*, int, int, int);
-template void unpack<uint16_t>(const uint8_t*, uint16_t*, int, int, int);
-template void unpack<uint32_t>(const uint8_t*, uint32_t*, int, int, int);
-template void unpack<uint64_t>(const uint8_t*, uint64_t*, int, int, int);
+template void unpack<bool>(const uint8_t*, bool*, const UnpackOptions&);
Review Comment:
I'm curious, why not put all the unpack-related APIs inside `arrow::internal::bpacking` as well? Does it cause too much code churn, or would it fail for other reasons?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
Review Comment:
I don't think there's something "above". AVX-512 would have `IsAvx2 = false`, right?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
Review Comment:
```suggestion
/// Whether we are compiling for the SSE2 or above in the SSE family (but not AVX2 / AVX-512)
```
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
Review Comment:
It seems we could try to fit more values pairs using swizzling, for example:
```
/// | ~ AAA | ~ CCC | ~ EEE | ~ GGG | ~III || ~ BBB | ~ DDD | ~ FFF | ~ HHH | ~JJJ |
```
We would be able to extract 10 values per swizzle instead of 4 (5 shifts per swizzle instead of 2).
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
+/// We therefore cannot end here and reapply the same kernel on the same input.
+/// Instead, we run another similar iteration to unpack E, F, G, H, this time taking into
+/// account that we have an extra offset of 12 bits (this will change all the constants
+/// used for swizzle and shift).
+///
+/// Note that in this example, we could have swizzled more than two values in each slot.
+/// In practice, there may be situations where a more complex algorithm could fit more
+/// shifts per swizzles, but that tend to not be the case when the SIMD register size
+/// increases.
+///
+/// @see KernelShape
+/// @see MediumKernelPlan
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = MediumKernelPlan<KerTraits, kOptions>::Build();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = decltype(kPlan)::Traits;
+ using unpacked_type = Traits::unpacked_type;
+ using uint_type = Traits::uint_type;
+ using simd_batch = Traits::simd_batch;
+ using simd_bytes = Traits::simd_bytes;
+ using arch_type = Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shift());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ safe_load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+/// Compile time dimensions of a Large kernel.
+///
+/// @see LargeKernel for explanation of the algorithm.
+struct LargeKernelPlanSize {
+ int reads_per_kernel_;
+ int unpacked_per_kernel_;
+
+ constexpr static LargeKernelPlanSize Build(const KernelShape& shape) {
+ const auto unpacked_per_kernel = std::lcm(shape.unpacked_per_simd(), 8);
+ const auto packed_bits_per_kernel = unpacked_per_kernel * shape.packed_bit_size();
+ const auto reads_per_kernel =
+ bit_util::CeilDiv(packed_bits_per_kernel, shape.simd_bit_size());
+
+ return {
+ .reads_per_kernel_ = static_cast<int>(reads_per_kernel),
+ .unpacked_per_kernel_ = unpacked_per_kernel,
+ };
+ }
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int unpacked_per_read() const {
+ return unpacked_per_kernel_ / reads_per_kernel_;
+ }
+ constexpr int unpacked_per_kernel() const { return unpacked_per_kernel_; }
+};
+
+/// Compile time constants of a Large kernel.
+///
+/// This contains most of the information about the large kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see LargeKernel for an explanation of the algorithm.
+template <typename KerTraits>
+struct LargeKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = LargeKernelPlanSize::Build(kShape);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shifts, kPlanSize.reads_per_kernel()>;
+
+ static constexpr LargeKernelPlan Build();
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kPlanSize.unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KerTraits>
+constexpr auto LargeKernelPlan<KerTraits>::Build() -> LargeKernelPlan<KerTraits> {
+ using Plan = LargeKernelPlan<KerTraits>;
+ using uint_type = Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ plan.low_swizzles.at(r) = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ // const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ // const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ const auto low_swizzle_val = packed_start_byte_in_read + b;
+ // Ensure we do not override previously defined values
+ assert(plan.low_swizzles.at(r).at(idx) == kUndefined);
+ assert(low_swizzle_val < kShape.simd_byte_size());
+
+ // Since kOverBytes is overly pessimistic, we may read an extra unneeded byte.
+ // On the last unpacked integer in the SIMD register, it may produce an out of
+ // bound swizzle. It would act the same as ``kUndefined`` but we make this
+ // explicit here.
+ // We need to stay pessimistic with ``kOverBytes`` to avoid having a mix of right
+ // and left shifts in the high swizzle.
+ auto high_swizzle_val = low_swizzle_val + kOverBytes;
+ if (high_swizzle_val >= kShape.simd_byte_size()) {
+ high_swizzle_val = kUndefined;
+ }
+ // Ensure we do not override previously defined values
+ assert(plan.high_swizzles.at(r).at(idx) == kUndefined);
+
+ plan.low_swizzles.at(r).at(idx) = static_cast<uint8_t>(low_swizzle_val);
+ plan.high_swizzles.at(r).at(idx) = static_cast<uint8_t>(high_swizzle_val);
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes create a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ assert(packed_start_bit % 8 == 0);
+
+ plan.mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for large sized inputs, where packed values spread over more bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 13 bits, extracted to
+/// uint16_t, using a fictive 32 bits SIMD size.
+/// The input register is as follow, where 13 consecutive letters represent the 13 bits of
+/// a packed value in little endian representation.
+///
+/// |AAAAAAAA|AAAAABBB||BBBBBBBB|BBCCCCCC|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 32/16 = 2 unpacked values per output register.
+/// Since 13-bit values can spread over 3 bytes (at worst bit alignment) but we're
+/// unpacking to uint16_t (2 bytes).
+///
+/// The core idea is to perform two separate swizzles: one for the "low" bits and one
+/// for the "high" bits of each value, then combine them with shifts and OR operations.
+///
+/// | Low bits from A || Low bits from B |
+/// low: |AAAAAAAA|AAAAABBB||AAAAABBB|BBBBBBBB|
+///
+/// | High bits frm A || High bits frm B |
+/// high: |AAAAABBB|BBBBBBBB||BBBBBBBB|BBCCCCCC|
+///
+/// We then apply different shifts to align the values bits. The low values are
+/// right-shifted by the bit offset, and the high values are left-shifted to fill the gap.
+/// Note that because of the figure is little endian, right (>>) and left (<<) shift
+/// operators actually shifts the bits representation in the oppostite directions.
+///
+/// | Align A w. >> 0 || Align B w. >> 5 |
+/// low: |AAAAAAAA|AAAAABBB||BBBBBBBBB|BB00000|
+///
+/// | Align A w. << 8 || Align B w. << 3 |
+/// high: |00000000|AAAAABBB||000BBBBB|BBBBBCCC|
+///
+/// We then proceed to merge both inputs. The idea is to mask out irrelevant bits from the
+/// high and low parts and merge with bitwise OR. However, in practice we can merge first
+/// and apply the mask only once afterwards (because we shifted zeros in). Below is the
+/// bitwise OR of both parts:
+///
+/// |AAAAAAAA|AAAAABBB||BBBBBBBBB|BB00000|
+/// & |00000000|AAAAABBB||000BBBBB|BBBBBCCC|
Review Comment:
You wrote `&` but it's `|`, right?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
+/// We therefore cannot end here and reapply the same kernel on the same input.
+/// Instead, we run another similar iteration to unpack E, F, G, H, this time taking into
+/// account that we have an extra offset of 12 bits (this will change all the constants
+/// used for swizzle and shift).
+///
+/// Note that in this example, we could have swizzled more than two values in each slot.
+/// In practice, there may be situations where a more complex algorithm could fit more
+/// shifts per swizzles, but that tend to not be the case when the SIMD register size
+/// increases.
Review Comment:
As a followup task, would be interesting to investigate in which (bit width, SIMD size) combinations a better algorithm could spare some swizzles.
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
Review Comment:
Do we want to assert that `kOffset < sizeof(ToInt) / sizeof(Int)`?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
Review Comment:
B, C, D were not byte aligned either, so I don't understand what makes E special.
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
Review Comment:
I notice the two 32-bit halves are slightly different but I'm not sure why, since we only care about keeping AAA, CCC, BBB and DDD. Why the difference between CDDDEEEF (first) and FFGGGHHH (second), for example?
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
+/// We therefore cannot end here and reapply the same kernel on the same input.
+/// Instead, we run another similar iteration to unpack E, F, G, H, this time taking into
+/// account that we have an extra offset of 12 bits (this will change all the constants
+/// used for swizzle and shift).
+///
+/// Note that in this example, we could have swizzled more than two values in each slot.
Review Comment:
Hence my question above :)
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,1106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This is a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <concepts>
+#include <cstdint>
+#include <cstring>
+#include <limits>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal::bpacking {
+
+template <typename T, std::size_t N>
+constexpr std::array<T, N> BuildConstantArray(T val) {
+ std::array<T, N> out = {};
+ for (auto& v : out) {
+ v = val;
+ }
+ return out;
+}
+
+template <typename Arr>
+constexpr Arr BuildConstantArrayLike(typename Arr::value_type val) {
+ return BuildConstantArray<typename Arr::value_type, std::tuple_size_v<Arr>>(val);
+}
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <std::unsigned_integral Uint, typename Arch>
+xsimd::batch<uint8_t, Arch> load_val_as(const uint8_t* in) {
+ const Uint val = util::SafeLoadAs<Uint>(in);
+ const auto batch = xsimd::batch<Uint, Arch>(val);
+ return xsimd::bitwise_cast<uint8_t>(batch);
+}
+
+template <int kBytes, typename Arch>
+xsimd::batch<uint8_t, Arch> safe_load_bytes(const uint8_t* in) {
+ if constexpr (kBytes <= sizeof(uint64_t)) {
+ return load_val_as<SizedUint<kBytes>, Arch>(in);
+ }
+ using simd_bytes = xsimd::batch<uint8_t, Arch>;
+ return simd_bytes::load_unaligned(in);
+}
+
+template <std::integral Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+/// Select indices in a batch constant according to a given stride.
+///
+/// Given a ``xsimd::batch_constant`` over a given integer type ``Int``, return a new
+/// batch constant over a larger integer type ``ToInt`` that select one of every "stride"
+/// element from the input.
+/// Because batch constants can only contain a predetermined number of values, the
+/// "stride" is determined as the ratio of the sizes of the output and input integer
+/// types.
+/// The ``kOffset`` is used to determine which sequence of values picked by the stride.
+/// Equivalently, it is the index of the first value selected.
+///
+/// For instance given an input with the following values
+/// |0|1|2|3|4|5|6|7|
+/// and a stride of 2 (e.g. sizeof(uint16_t)/sizeof(uint8_t)), an offset of 0 would
+/// return the values:
+/// |0|2|4|6|
+/// while an offset of 1 would return the values:
+/// |1|3|5|7|
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+/// Whether we are compiling for the SSE2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+/// Whether we are compiling for the AVX2 or above in the SSE family.
+template <typename Arch>
+constexpr bool IsAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kIsSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kIsAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kIsSse2 || kIsAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts) {
+ constexpr bool kIsSse2 = IsSse2<Arch>;
+ constexpr bool kIsAvx2 = IsAvx2<Arch>;
+ static_assert(
+ !(kIsSse2 && kIsAvx2),
+ "In xsimd, an x86 arch is either part of the SSE family or of the AVX family,"
+ "not both. If this check fails, it means the assumptions made here to detect SSE "
+ "and AVX are out of date.");
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Architecture for which there is no variable right shift but a larger fallback exists.
+ /// TODO(xsimd) Tracking for Avx2 in https://github.com/xtensor-stack/xsimd/pull/1220
+ if constexpr (kIsAvx2 && (IntSize == sizeof(uint8_t) || IntSize == sizeof(uint16_t))) {
+ using twice_uint = SizedUint<2 * IntSize>;
+
+ const auto batch2 = xsimd::bitwise_cast<twice_uint>(batch);
+
+ constexpr auto kShifts0 = select_stride<twice_uint, 0>(shifts);
+ constexpr auto kMask0 = bit_util::LeastSignificantBitMask<twice_uint>(8 * IntSize);
+ const auto shifted0 = right_shift_by_excess(batch2 & kMask0, kShifts0);
+
+ constexpr auto kShifts1 = select_stride<twice_uint, 1>(shifts);
+ constexpr auto kMask1 = kMask0 << (8 * IntSize);
+ const auto shifted1 = right_shift_by_excess(batch2, kShifts1) & kMask1;
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ // These conditions are the ones matched in `left_shift`, i.e. the ones where variable
+ // shift right will not be available but a left shift (fallback) exists.
+ if constexpr (kIsSse2 && (IntSize != sizeof(uint64_t))) {
+ constexpr Int kMaxRShift = max_value(std::array{kShifts...});
+
+ constexpr auto kLShifts =
+ xsimd::make_batch_constant<Int, kMaxRShift, Arch>() - shifts;
+
+ return xsimd::bitwise_rshift<kMaxRShift>(left_shift(batch, kLShifts));
+ }
+
+ return batch >> shifts;
+}
+
+/****************************
+ * Properties of a kernel *
+ ****************************/
+
+/// Compute whether packed values may overflow SIMD register.
+///
+/// Due to packing misalignment, we may not be able to fit in a SIMD register as many
+/// packed values as unpacked values.
+/// This happens because of the packing bit alignment creates spare bits we are forced to
+/// read when reading whole bytes. This is mostly known to be the case with 63 bits values
+/// in a 128 bit SIMD register.
+///
+/// @see KernelShape
+/// @see PackedMaxSpreadBytes
+constexpr bool PackedIsOversizedForSimd(int simd_bit_size, int unpacked_bit_size,
+ int packed_bit_size) {
+ const int unpacked_per_simd = simd_bit_size / unpacked_bit_size;
+
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ return (simd_bit_size - bit_offset) / packed_bit_size;
+ };
+
+ int packed_start_bit = 0;
+ // Try all possible bit offsets for this packed size
+ do {
+ int packed_per_read = packed_per_read_for_offset(packed_start_bit % 8);
+ if (packed_per_read < unpacked_per_simd) {
+ // A single SIMD batch isn't enough to unpack from this bit offset,
+ // so the "medium" model cannot work.
+ return true;
+ }
+ packed_start_bit += unpacked_per_simd * packed_bit_size;
+ } while (packed_start_bit % 8 != 0);
+
+ return false;
+}
+
+/// Different sizes of a given kernel.
+///
+/// When integers are bit-packed, they can spread over multiple bytes.
+/// For instance, integers packed over three bits quickly spread over two bytes (on the
+/// value `C` below) despite three bits being much smaller than a single byte.
+///
+/// ```
+/// |A|A|A|B|B|B|C|C| |C|D|D|D|E|E|E|F| ...
+/// ```
+///
+/// When the spread is smaller or equal to the unsigned integer to unpack to, we classify
+/// if as "medium".
+///
+/// When it is strictly larger, it classifies as "large".
+///
+/// On rare occasions, due to offsets in reading a subsequent batch, we may not even be
+/// able to read as many packed values as we should extract in a batch. We classify this
+/// as "oversized". For instance unpacking 63 bits values to uint64_t on 128 bits SIMD
+/// register (possibly the only such case), we need to write 2 uint64_t on each output
+/// register that we flush to memory. The first two value spread over bits [0, 126[,
+/// fitting in 16 bytes (the full SIMD register). However, the next two values on input
+/// bits [126, 252[, are not byte aligned and would require reading the whole [120, 256[
+/// range, or 17 bytes. We would need to read more than one SIMD register to create
+/// a single output register with two values. Such pathological cases are not handled with
+/// SIMD but rather fully falling back to the exact scalar loop.
+///
+/// There is currently no "small" classification, as medium kernels can handle even 1-bit
+/// packed values. Future work may investigate a dedicated kernel for when a single byte
+/// can fit more values than is being unpack in a single register. For instance, unpacking
+/// 1 bit values to uint32_t on 128 bits SIMD register, we can fit 8 values in a byte, or
+/// twice the number of uint32_t that fit in an SIMD register. This can be use to further
+/// reduce the number of swizzles done in a kernel run. Other optimizations of interest
+/// for small input sizes on BMI2 is to use ``_pdep_u64`` directly to perform unpacking.
+struct KernelShape {
+ const int simd_bit_size_;
+ const int unpacked_bit_size_;
+ const int packed_bit_size_;
+ const int packed_max_spread_bytes_ = PackedMaxSpreadBytes(packed_bit_size_, 0);
+ const bool is_oversized_ =
+ PackedIsOversizedForSimd(simd_bit_size_, unpacked_bit_size_, packed_bit_size_);
+
+ constexpr bool is_medium() const {
+ return packed_max_spread_bytes() <= unpacked_byte_size();
+ }
+ constexpr bool is_large() const { return !is_medium() && !is_oversized(); }
+ constexpr bool is_oversized() const { return is_oversized_; }
+
+ /// Properties of an SIMD batch
+ constexpr int simd_bit_size() const { return simd_bit_size_; }
+ constexpr int simd_byte_size() const { return simd_bit_size_ / 8; }
+
+ /// Properties of the unpacked integers
+ constexpr int unpacked_bit_size() const { return unpacked_bit_size_; }
+ constexpr int unpacked_byte_size() const { return unpacked_bit_size_ / 8; }
+ constexpr int unpacked_per_simd() const { return simd_bit_size_ / unpacked_bit_size_; }
+
+ /// Properties of the packed integers
+ constexpr int packed_bit_size() const { return packed_bit_size_; }
+ constexpr int packed_max_spread_bytes() const { return packed_max_spread_bytes_; }
+};
+
+/// Packing all useful and derived information about a kernel in a single type.
+template <std::unsigned_integral UnpackedUint, int kPackedBitSize, int kSimdBitSize>
+struct KernelTraits {
+ static constexpr KernelShape kShape = {
+ .simd_bit_size_ = kSimdBitSize,
+ .unpacked_bit_size_ = 8 * sizeof(UnpackedUint),
+ .packed_bit_size_ = kPackedBitSize,
+ };
+
+ static_assert(kShape.simd_bit_size() % kShape.unpacked_bit_size() == 0);
+ static_assert(0 < kShape.packed_bit_size());
+ static_assert(kShape.packed_bit_size() < kShape.simd_bit_size());
+
+ using unpacked_type = UnpackedUint;
+ /// The integer type to work with, `unpacked_type` or an appropriate type for bool.
+ using uint_type = std::conditional_t<std::is_same_v<unpacked_type, bool>,
+ SizedUint<sizeof(bool)>, unpacked_type>;
+ using simd_batch = xsimd::make_sized_batch_t<uint_type, kShape.unpacked_per_simd()>;
+ using simd_bytes = xsimd::make_sized_batch_t<uint8_t, kShape.simd_byte_size()>;
+ using arch_type = simd_batch::arch_type;
+};
+
+/// Return similar kernel traits but with a different integer unpacking type.
+template <typename KerTraits, std::unsigned_integral Uint>
+using KernelTraitsWithUnpackUint = KernelTraits<Uint, KerTraits::kShape.packed_bit_size(),
+ KerTraits::kShape.simd_bit_size()>;
+
+/******************
+ * MediumKernel *
+ ******************/
+
+/// Compile time options of a Medium kernel.
+struct MediumKernelOptions {
+ /// An indicative limit on the number of values unpacked by the kernel.
+ /// This is a heuristic setting: other constraints such as alignment may not always make
+ /// small values feasibles. Must be a power of two.
+ int unpacked_per_kernel_limit_;
+};
+
+/// Compile time dimensions of a Medium kernel.
+///
+/// @see MediumKernel for explanation of the algorithm.
+struct MediumKernelPlanSize {
+ int reads_per_kernel_;
+ int swizzles_per_read_;
+ int shifts_per_swizzle_;
+
+ constexpr static MediumKernelPlanSize Build(const KernelShape& shape,
+ const MediumKernelOptions& options);
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int swizzles_per_read() const { return swizzles_per_read_; }
+ constexpr int swizzles_per_kernel() const {
+ return swizzles_per_read_ * reads_per_kernel();
+ }
+
+ constexpr int shifts_per_swizzle() const { return shifts_per_swizzle_; }
+ constexpr int shifts_per_read() const {
+ return shifts_per_swizzle_ * swizzles_per_read();
+ }
+ constexpr int shifts_per_kernel() const {
+ return shifts_per_read() * reads_per_kernel();
+ }
+};
+
+constexpr MediumKernelPlanSize MediumKernelPlanSize::Build(
+ const KernelShape& shape, const MediumKernelOptions& options) {
+ const int shifts_per_swizzle =
+ shape.unpacked_byte_size() / shape.packed_max_spread_bytes();
+
+ const int vals_per_swizzle = shifts_per_swizzle * shape.unpacked_per_simd();
+
+ // Using `unpacked_per_kernel_limit` to influence the number of swizzles per reads.
+ const auto packed_per_read_for_offset = [&](int bit_offset) -> int {
+ const int best = (shape.simd_bit_size() - bit_offset) / shape.packed_bit_size();
+ const int limit = options.unpacked_per_kernel_limit_;
+ return (best > limit) && (limit > 0) ? limit : best;
+ };
+
+ const auto swizzles_per_read_for_offset = [&](int bit_offset) -> int {
+ return packed_per_read_for_offset(bit_offset) / vals_per_swizzle;
+ };
+
+ // If after a whole swizzle reading iteration we fall byte-unaligned, the remaining
+ // iterations will start with an byte-unaligned first value, reducing the effective
+ // capacity of the SIMD batch.
+ // We must check that our read iteration size still works with subsequent misalignment
+ // by looping until aligned. This implies that we do the same number of swizzle for each
+ // read offset. This may be improved a bit for some bit width, but in practice there are
+ // very few such cases.
+ // One may think that this algorithm will drive the kernel minimum reading size to
+ // unreasonably large sizes, but in practice reading iterations increase by factors of 2
+ // and are quickly multiple of 8 (and aligned after the first one).
+ int swizzles_per_read = swizzles_per_read_for_offset(0);
+ int reads_per_kernel = 0;
+ int packed_start_bit = 0;
+ do {
+ int new_swizzles_per_read = swizzles_per_read_for_offset(packed_start_bit % 8);
+ if (new_swizzles_per_read < swizzles_per_read) {
+ swizzles_per_read = new_swizzles_per_read;
+ packed_start_bit = 0;
+ reads_per_kernel = 0;
+ }
+ int bits_per_read = swizzles_per_read * vals_per_swizzle * shape.packed_bit_size();
+ packed_start_bit += bits_per_read;
+ reads_per_kernel += 1;
+ } while (packed_start_bit % 8 != 0);
+
+ return {
+ .reads_per_kernel_ = reads_per_kernel,
+ .swizzles_per_read_ = swizzles_per_read,
+ .shifts_per_swizzle_ = shifts_per_swizzle,
+ };
+}
+
+/// Utility to map a desired number of bits to the smallest feasible size.
+///
+/// When needing to read @p bits_per_read, find the next possible size that can be read in
+/// one operation. This is used to adjust the number of input bytes to kernel reads.
+/// For instance if a kernel plan needs to read 47 bits, instead of reading a whole SIMD
+/// register of data (which would restrict it to larger input memory buffers), this
+/// function advise kernel plans to read only read 64 bits.
+/// This limits restrictions set by the plan on the input memory reads built to avoid
+/// reading overflow.
+constexpr int adjust_bytes_per_read(int bits_per_read, int simd_byte_size) {
+ if (bits_per_read <= static_cast<int>(8 * sizeof(uint32_t))) {
+ return sizeof(uint32_t);
+ } else if (bits_per_read <= static_cast<int>(8 * sizeof(uint64_t))) {
+ return sizeof(uint64_t);
+ }
+ return simd_byte_size;
+}
+
+/// Compile time constants of a Medium kernel.
+///
+/// This contains most of the information about the medium kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see MediumKernel for an explanation of the algorithm.
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = MediumKernelPlanSize::Build(kShape, kOptions);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerRead = std::array<Swizzle, kPlanSize.swizzles_per_read()>;
+ using SwizzlesPerKernel = std::array<SwizzlesPerRead, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShiftsPerSwizzle = std::array<Shifts, kPlanSize.shifts_per_swizzle()>;
+ using ShiftsPerRead = std::array<ShiftsPerSwizzle, kPlanSize.swizzles_per_read()>;
+ using ShiftsPerKernel = std::array<ShiftsPerRead, kPlanSize.reads_per_kernel()>;
+
+ constexpr static MediumKernelPlan Build();
+
+ static constexpr int unpacked_per_shift() { return kShape.unpacked_per_simd(); }
+ static constexpr int unpacked_per_swizzle() {
+ return unpacked_per_shift() * kPlanSize.shifts_per_swizzle();
+ }
+ static constexpr int unpacked_per_read() {
+ return unpacked_per_swizzle() * kPlanSize.swizzles_per_read();
+ }
+ static constexpr int unpacked_per_kernel() {
+ return unpacked_per_read() * kPlanSize.reads_per_kernel();
+ }
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel swizzles;
+ ShiftsPerKernel shifts;
+ uint_type mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+};
+
+template <typename KerTraits, MediumKernelOptions kOptions>
+constexpr auto MediumKernelPlan<KerTraits, kOptions>::Build()
+ -> MediumKernelPlan<KerTraits, kOptions> {
+ using Plan = MediumKernelPlan<KerTraits, kOptions>;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_medium());
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ for (int sw = 0; sw < kPlanSize.swizzles_per_read(); ++sw) {
+ auto& swizzles = plan.swizzles.at(r).at(sw);
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ swizzles = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ for (int sh = 0; sh < kPlanSize.shifts_per_swizzle(); ++sh) {
+ const int sh_offset_bytes = sh * kShape.packed_max_spread_bytes();
+ assert(sh_offset_bytes < kShape.simd_byte_size());
+ const int sh_offset_bits = 8 * sh_offset_bytes;
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ const int u_offset_byte = u * kShape.unpacked_byte_size();
+ assert(u_offset_byte < kShape.simd_byte_size());
+ const int sw_offset_byte = sh_offset_bytes + u_offset_byte;
+ assert(sw_offset_byte < kShape.simd_byte_size());
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over the multiple bytes needed for current values
+ const int spread = packed_end_byte - packed_start_byte;
+ for (int b = 0; b < spread; ++b) {
+ const auto swizzle_val = static_cast<uint8_t>(packed_start_byte_in_read + b);
+ // Ensure we do not override previously defined value
+ assert(swizzles.at(sw_offset_byte + b) == kUndefined);
+ // Ensure value is within permutation bounds
+ assert(swizzle_val < kShape.simd_byte_size());
+ swizzles.at(sw_offset_byte + b) = swizzle_val;
+ }
+
+ // A given swizzle may contain data for multiple shifts. They are stored in
+ // different shift arrays (indexed by ``sh``).
+ plan.shifts.at(r).at(sw).at(sh).at(u) = sh_offset_bits + packed_start_bit % 8;
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ }
+ }
+
+ assert(packed_start_bit % 8 == 0);
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for medium sized inputs where packed values spread over fewer bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 3 bits, extracted to
+/// uint32_t, using a fictive 64 bits SIMD size.
+/// The input register is as follow, where 3 consecutive letter represents the 3 bits of a
+/// packed value in little endian representation.
+///
+/// |AAABBBCC|CDDDEEEF|FFGGGHHH|IIIJJJKK||KLLLMMMN|NNOOOPPP|QQQRRRSS|STTTUUUV|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 64/32 = 2 unpacked values per output register.
+/// The first step is to put the values in the 32 bit slots corresponding to the unpacked
+/// values using a swizzle (shuffle/indexing) operation. The spread for 3 bit value is of
+/// 2 bytes (at worst bit alignment) so we move 2 bytes at the time.
+/// We care about creating a register as follows:
+///
+/// | ~ Val AAA anywhere here || ~ Val BBB anywhere here |
+///
+/// Since we have unused bytes in this register, we will also use them to fit the next
+/// values:
+///
+/// | ~ Val AAA | ~ Val CCC || ~ Val BBB | ~ Val DDD |
+///
+/// Resulting in the following swizzled register.
+///
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||AAABBBCC|CDDDEEEF|CDDDEEEF|FFGGGHHH|
+///
+/// Now we will apply shift and mask operations to properly align the values.
+/// Shifting has a different value for each uint32_t in the SIMD register.
+/// Note that because of the figure is little endian, right shift operator >> actually
+/// shifts the bits representation, to the left.
+///
+/// | Align AAA with >> 0 || Align BBB with >> 3 |
+/// |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+///
+/// Masking (bitewise AND) uses the same mask for all uint32_t
+///
+/// |11100000000000000000000000000000000||11100000000000000000000000000000000|
+/// & |AAABBBCC|CDDDEEEF|AAABBBCC|CDDDEEEF||BBBCCCDD|DEEEFCDD|DEEEFFFG|GGHHH000|
+/// = |AAA00000|00000000|00000000|00000000||BBB00000|00000000|00000000|00000000|
+///
+/// We have created a SIMD register with the first two unpacked values A and B that we can
+/// write to memory.
+/// Now we can do the same with C and D without having to go to another swizzle (doing
+/// multiple shifts per swizzle).
+/// The next value to unpack is E, which starts on bit 12 (not byte aligned).
+/// We therefore cannot end here and reapply the same kernel on the same input.
+/// Instead, we run another similar iteration to unpack E, F, G, H, this time taking into
+/// account that we have an extra offset of 12 bits (this will change all the constants
+/// used for swizzle and shift).
+///
+/// Note that in this example, we could have swizzled more than two values in each slot.
+/// In practice, there may be situations where a more complex algorithm could fit more
+/// shifts per swizzles, but that tend to not be the case when the SIMD register size
+/// increases.
+///
+/// @see KernelShape
+/// @see MediumKernelPlan
+template <typename KerTraits, MediumKernelOptions kOptions>
+struct MediumKernel {
+ static constexpr auto kPlan = MediumKernelPlan<KerTraits, kOptions>::Build();
+ static constexpr auto kPlanSize = kPlan.kPlanSize;
+ static constexpr auto kShape = kPlan.kShape;
+ using Traits = decltype(kPlan)::Traits;
+ using unpacked_type = Traits::unpacked_type;
+ using uint_type = Traits::uint_type;
+ using simd_batch = Traits::simd_batch;
+ using simd_bytes = Traits::simd_bytes;
+ using arch_type = Traits::arch_type;
+
+ static constexpr int kValuesUnpacked = kPlan.unpacked_per_kernel();
+ static constexpr int kBytesRead = kPlan.total_bytes_read();
+
+ template <int kReadIdx, int kSwizzleIdx, int kShiftIdx>
+ static void unpack_one_shift_impl(const simd_batch& words, unpacked_type* out) {
+ constexpr auto kRightShiftsArr =
+ kPlan.shifts.at(kReadIdx).at(kSwizzleIdx).at(kShiftIdx);
+ constexpr auto kRightShifts = array_to_batch_constant<kRightShiftsArr, arch_type>();
+ constexpr auto kMask = kPlan.mask;
+ constexpr auto kOutOffset = (kReadIdx * kPlan.unpacked_per_read() +
+ kSwizzleIdx * kPlan.unpacked_per_swizzle() +
+ kShiftIdx * kPlan.unpacked_per_shift());
+
+ // Intel x86-64 does not have variable right shifts before AVX2.
+ // We know the packed value can safely be left shifted up to the largest offset so we
+ // can use the fallback on these platforms.
+ const auto shifted = right_shift_by_excess(words, kRightShifts);
+ const auto vals = shifted & kMask;
+ if constexpr (std::is_same_v<unpacked_type, bool>) {
+ const xsimd::batch_bool<uint_type, arch_type> bools = vals != 0;
+ bools.store_unaligned(out + kOutOffset);
+ } else {
+ vals.store_unaligned(out + kOutOffset);
+ }
+ }
+
+ template <int kReadIdx, int kSwizzleIdx, int... kShiftIds>
+ static void unpack_one_swizzle_impl(const simd_bytes& bytes, unpacked_type* out,
+ std::integer_sequence<int, kShiftIds...>) {
+ constexpr auto kSwizzlesArr = kPlan.swizzles.at(kReadIdx).at(kSwizzleIdx);
+ constexpr auto kSwizzles = array_to_batch_constant<kSwizzlesArr, arch_type>();
+
+ const auto swizzled = xsimd::swizzle(bytes, kSwizzles);
+ const auto words = xsimd::bitwise_cast<uint_type>(swizzled);
+ (unpack_one_shift_impl<kReadIdx, kSwizzleIdx, kShiftIds>(words, out), ...);
+ }
+
+ template <int kReadIdx, int... kSwizzleIds>
+ static void unpack_one_read_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kSwizzleIds...>) {
+ using ShiftSeq = std::make_integer_sequence<int, kPlanSize.shifts_per_swizzle()>;
+ const auto bytes =
+ safe_load_bytes<kPlan.bytes_per_read(), arch_type>(in + kPlan.reads.at(kReadIdx));
+ (unpack_one_swizzle_impl<kReadIdx, kSwizzleIds>(bytes, out, ShiftSeq{}), ...);
+ }
+
+ template <int... kReadIds>
+ static void unpack_all_impl(const uint8_t* in, unpacked_type* out,
+ std::integer_sequence<int, kReadIds...>) {
+ using SwizzleSeq = std::make_integer_sequence<int, kPlanSize.swizzles_per_read()>;
+ (unpack_one_read_impl<kReadIds>(in, out, SwizzleSeq{}), ...);
+ }
+
+ static const uint8_t* unpack(const uint8_t* in, unpacked_type* out) {
+ using ReadSeq = std::make_integer_sequence<int, kPlanSize.reads_per_kernel()>;
+ unpack_all_impl(in, out, ReadSeq{});
+ return in + (kPlan.unpacked_per_kernel() * kShape.packed_bit_size()) / 8;
+ }
+};
+
+/*****************
+ * LargeKernel *
+ *****************/
+
+/// Compile time dimensions of a Large kernel.
+///
+/// @see LargeKernel for explanation of the algorithm.
+struct LargeKernelPlanSize {
+ int reads_per_kernel_;
+ int unpacked_per_kernel_;
+
+ constexpr static LargeKernelPlanSize Build(const KernelShape& shape) {
+ const auto unpacked_per_kernel = std::lcm(shape.unpacked_per_simd(), 8);
+ const auto packed_bits_per_kernel = unpacked_per_kernel * shape.packed_bit_size();
+ const auto reads_per_kernel =
+ bit_util::CeilDiv(packed_bits_per_kernel, shape.simd_bit_size());
+
+ return {
+ .reads_per_kernel_ = static_cast<int>(reads_per_kernel),
+ .unpacked_per_kernel_ = unpacked_per_kernel,
+ };
+ }
+
+ constexpr int reads_per_kernel() const { return reads_per_kernel_; }
+
+ constexpr int unpacked_per_read() const {
+ return unpacked_per_kernel_ / reads_per_kernel_;
+ }
+ constexpr int unpacked_per_kernel() const { return unpacked_per_kernel_; }
+};
+
+/// Compile time constants of a Large kernel.
+///
+/// This contains most of the information about the large kernel algorithm stored
+/// as index offsets and other constants (shift amounts...).
+///
+/// @see LargeKernel for an explanation of the algorithm.
+template <typename KerTraits>
+struct LargeKernelPlan {
+ using Traits = KerTraits;
+ using uint_type = Traits::uint_type;
+ static constexpr auto kShape = Traits::kShape;
+ static constexpr auto kPlanSize = LargeKernelPlanSize::Build(kShape);
+
+ /// Array of byte offsets relative to the Kernel input memory address.
+ using ReadsPerKernel = std::array<int, kPlanSize.reads_per_kernel()>;
+
+ /// Array of indices used for byte swizzle/shuffle in an SIMD register.
+ using Swizzle = std::array<uint8_t, kShape.simd_byte_size()>;
+ using SwizzlesPerKernel = std::array<Swizzle, kPlanSize.reads_per_kernel()>;
+
+ /// Array of integer shifts for each unpacked integer in an SIMD register.
+ using Shifts = std::array<uint_type, kShape.unpacked_per_simd()>;
+ using ShitsPerKernel = std::array<Shifts, kPlanSize.reads_per_kernel()>;
+
+ static constexpr LargeKernelPlan Build();
+
+ static constexpr int bytes_per_read() {
+ const auto bits_per_read = kPlanSize.unpacked_per_read() * kShape.packed_bit_size();
+ return adjust_bytes_per_read(bits_per_read, kShape.simd_byte_size());
+ }
+
+ constexpr int total_bytes_read() const { return reads.back() + bytes_per_read(); }
+
+ ReadsPerKernel reads;
+ SwizzlesPerKernel low_swizzles;
+ SwizzlesPerKernel high_swizzles;
+ ShitsPerKernel low_rshifts;
+ ShitsPerKernel high_lshifts;
+ uint_type mask;
+};
+
+template <typename KerTraits>
+constexpr auto LargeKernelPlan<KerTraits>::Build() -> LargeKernelPlan<KerTraits> {
+ using Plan = LargeKernelPlan<KerTraits>;
+ using uint_type = Plan::Traits::uint_type;
+ constexpr auto kShape = Plan::kShape;
+ constexpr auto kPlanSize = Plan::kPlanSize;
+ static_assert(kShape.is_large());
+ constexpr int kOverBytes =
+ kShape.packed_max_spread_bytes() - kShape.unpacked_byte_size();
+
+ Plan plan = {};
+
+ int packed_start_bit = 0;
+ for (int r = 0; r < kPlanSize.reads_per_kernel(); ++r) {
+ const int read_start_byte = packed_start_bit / 8;
+ plan.reads.at(r) = read_start_byte;
+
+ // Not all swizzle values are defined. This is not an issue as these bits do not
+ // influence the input (they will be masked away).
+ constexpr uint8_t kUndefined = std::numeric_limits<uint8_t>::max();
+ plan.low_swizzles.at(r) = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+ plan.high_swizzles.at(r) = BuildConstantArrayLike<Plan::Swizzle>(kUndefined);
+
+ // Looping over unpacked values in the output registers
+ for (int u = 0; u < kShape.unpacked_per_simd(); ++u) {
+ // We compute where the corresponding packed value will be in the input
+ const int packed_start_byte = packed_start_bit / 8;
+ const int packed_start_byte_in_read = packed_start_byte - read_start_byte;
+ assert(0 <= packed_start_byte_in_read);
+ assert(packed_start_byte_in_read < kShape.simd_byte_size());
+ // const int packed_end_bit = packed_start_bit + kShape.packed_bit_size();
+ // const int packed_end_byte = (packed_end_bit - 1) / 8 + 1;
+
+ // Looping over maximum number of bytes that can fit a value
+ // We fill more than necessary in the high swizzle because in the absence of
+ // variable right shifts, we will erase some bits from the low sizzled values.
+ for (int b = 0; b < kShape.unpacked_byte_size(); ++b) {
+ const auto idx = u * kShape.unpacked_byte_size() + b;
+ const auto low_swizzle_val = packed_start_byte_in_read + b;
+ // Ensure we do not override previously defined values
+ assert(plan.low_swizzles.at(r).at(idx) == kUndefined);
+ assert(low_swizzle_val < kShape.simd_byte_size());
+
+ // Since kOverBytes is overly pessimistic, we may read an extra unneeded byte.
+ // On the last unpacked integer in the SIMD register, it may produce an out of
+ // bound swizzle. It would act the same as ``kUndefined`` but we make this
+ // explicit here.
+ // We need to stay pessimistic with ``kOverBytes`` to avoid having a mix of right
+ // and left shifts in the high swizzle.
+ auto high_swizzle_val = low_swizzle_val + kOverBytes;
+ if (high_swizzle_val >= kShape.simd_byte_size()) {
+ high_swizzle_val = kUndefined;
+ }
+ // Ensure we do not override previously defined values
+ assert(plan.high_swizzles.at(r).at(idx) == kUndefined);
+
+ plan.low_swizzles.at(r).at(idx) = static_cast<uint8_t>(low_swizzle_val);
+ plan.high_swizzles.at(r).at(idx) = static_cast<uint8_t>(high_swizzle_val);
+ }
+
+ // low and high swizzles need to be rshifted but the oversized bytes create a
+ // larger lshift for high values.
+ plan.low_rshifts.at(r).at(u) = packed_start_bit % 8;
+ plan.high_lshifts.at(r).at(u) = 8 * kOverBytes - (packed_start_bit % 8);
+
+ packed_start_bit += kShape.packed_bit_size();
+ }
+ }
+ assert(packed_start_bit % 8 == 0);
+
+ plan.mask = bit_util::LeastSignificantBitMask<uint_type>(kShape.packed_bit_size());
+
+ return plan;
+}
+
+/// Unpack a fixed number of packed integer using SIMD operations.
+///
+/// This is only valid for large sized inputs, where packed values spread over more bytes
+/// than the unpacked integer size.
+/// All dimensions and constants are computed at compile time to reduce run-time
+/// computations and improve xsimd fallback possibilities (with
+/// ``xsimd::batch_constant``).
+///
+/// Let us give an example of the algorithm for values packed over 13 bits, extracted to
+/// uint16_t, using a fictive 32 bits SIMD size.
+/// The input register is as follow, where 13 consecutive letters represent the 13 bits of
+/// a packed value in little endian representation.
+///
+/// |AAAAAAAA|AAAAABBB||BBBBBBBB|BBCCCCCC|
+///
+/// The algorithm generates SIMD registers of unpacked values ready to be written to
+/// memory. Here we can fit 32/16 = 2 unpacked values per output register.
+/// Since 13-bit values can spread over 3 bytes (at worst bit alignment) but we're
+/// unpacking to uint16_t (2 bytes).
+///
+/// The core idea is to perform two separate swizzles: one for the "low" bits and one
+/// for the "high" bits of each value, then combine them with shifts and OR operations.
+///
+/// | Low bits from A || Low bits from B |
+/// low: |AAAAAAAA|AAAAABBB||AAAAABBB|BBBBBBBB|
+///
+/// | High bits frm A || High bits frm B |
+/// high: |AAAAABBB|BBBBBBBB||BBBBBBBB|BBCCCCCC|
+///
+/// We then apply different shifts to align the values bits. The low values are
+/// right-shifted by the bit offset, and the high values are left-shifted to fill the gap.
+/// Note that because of the figure is little endian, right (>>) and left (<<) shift
+/// operators actually shifts the bits representation in the oppostite directions.
Review Comment:
```suggestion
/// operators actually shift the bits representation in the opposite directions.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #47994:
URL: https://github.com/apache/arrow/pull/47994#discussion_r2783173930
##########
cpp/src/arrow/util/bpacking_simd_kernel_internal.h:
##########
@@ -0,0 +1,790 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/// Simd integer unpacking kernels, that is small functions that efficiently operate over
+/// a fixed input size.
+///
+/// This a generalization of the algorithm from Daniel Lemire and Leonid Boytsov,
+/// Decoding billions of integers per second through vectorization, Software Practice &
+/// Experience 45 (1), 2015.
+/// http://arxiv.org/abs/1209.2137
+/// https://github.com/fast-pack/LittleIntPacker/blob/master/src/horizontalpacking32.c
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+#include <cstring>
+#include <numeric>
+#include <utility>
+
+#include <xsimd/xsimd.hpp>
+
+#include "arrow/util/bit_util.h"
+#include "arrow/util/bpacking_dispatch_internal.h"
+#include "arrow/util/type_traits.h"
+
+namespace arrow::internal {
+
+/*********************
+ * xsimd utilities *
+ *********************/
+
+/// Simple constexpr maximum element suited for non empty arrays.
+template <typename T, std::size_t N>
+constexpr T max_value(const std::array<T, N>& arr) {
+ static_assert(N > 0);
+ T out = 0;
+ for (const T& v : arr) {
+ if (v > out) {
+ out = v;
+ }
+ }
+ return out;
+}
+
+template <std::array kArr, typename Arch, std::size_t... Is>
+constexpr auto array_to_batch_constant_impl(std::index_sequence<Is...>) {
+ using Array = std::decay_t<decltype(kArr)>;
+ using value_type = typename Array::value_type;
+
+ return xsimd::batch_constant<value_type, Arch, kArr[Is]...>{};
+}
+
+/// Make a ``xsimd::batch_constant`` from a static constexpr array.
+template <std::array kArr, typename Arch>
+constexpr auto array_to_batch_constant() {
+ return array_to_batch_constant_impl<kArr, Arch>(
+ std::make_index_sequence<kArr.size()>());
+}
+
+template <typename Int, int kOffset, int kLength, typename Arr>
+constexpr auto select_stride_impl(Arr shifts) {
+ std::array<Int, shifts.size() / kLength> out{};
+ for (std::size_t i = 0; i < out.size(); ++i) {
+ out[i] = shifts[kLength * i + kOffset];
+ }
+ return out;
+}
+
+template <typename ToInt, int kOffset, typename Int, typename Arch, Int... kShifts>
+constexpr auto select_stride(xsimd::batch_constant<Int, Arch, kShifts...>) {
+ constexpr auto kStridesArr =
+ select_stride_impl<ToInt, kOffset, sizeof(ToInt) / sizeof(Int)>(
+ std::array{kShifts...});
+ return array_to_batch_constant<kStridesArr, Arch>();
+}
+
+template <typename Arch>
+constexpr bool HasSse2 = std::is_base_of_v<xsimd::sse2, Arch>;
+
+template <typename Arch>
+constexpr bool HasAvx2 = std::is_base_of_v<xsimd::avx2, Arch>;
+
+/// Wrapper around ``xsimd::bitwise_lshift`` with optimizations for non implemented sizes.
+//
+// We replace the variable left shift by a variable multiply with a power of two.
+//
+// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+// integers per second through vectorization, Software Practice & Experience 45 (1), 2015.
+// http://arxiv.org/abs/1209.2137
+//
+/// TODO(xsimd) Tracking in https://github.com/xtensor-stack/xsimd/pull/1220
+template <typename Arch, typename Int, Int... kShifts>
+auto left_shift(const xsimd::batch<Int, Arch>& batch,
+ xsimd::batch_constant<Int, Arch, kShifts...> shifts)
+ -> xsimd::batch<Int, Arch> {
+ constexpr bool kHasSse2 = HasSse2<Arch>;
+ constexpr bool kHasAvx2 = HasAvx2<Arch>;
+ static_assert(!(kHasSse2 && kHasAvx2), "The hierarchy are different in xsimd");
+
+ constexpr auto kMults = xsimd::make_batch_constant<Int, 1, Arch>() << shifts;
+
+ constexpr auto IntSize = sizeof(Int);
+
+ // Sizes and architecture for which there is no variable left shift and there is a
+ // multiplication
+ if constexpr ( //
+ (kHasSse2 && (IntSize == sizeof(uint16_t) || IntSize == sizeof(uint32_t))) //
+ || (kHasAvx2 && (IntSize == sizeof(uint16_t))) //
+ ) {
+ return batch * kMults;
+ }
+
+ // Architecture for which there is no variable left shift on uint8_t but a fallback
+ // exists for uint16_t.
+ if constexpr ((kHasSse2 || kHasAvx2) && (IntSize == sizeof(uint8_t))) {
+ const auto batch16 = xsimd::bitwise_cast<uint16_t>(batch);
+
+ constexpr auto kShifts0 = select_stride<uint16_t, 0>(shifts);
+ const auto shifted0 = left_shift(batch16, kShifts0) & 0x00FF;
+
+ constexpr auto kShifts1 = select_stride<uint16_t, 1>(shifts);
+ const auto shifted1 = left_shift(batch16 & 0xFF00, kShifts1);
+
+ return xsimd::bitwise_cast<Int>(shifted0 | shifted1);
+ }
+
+ return batch << shifts;
+}
+
+/// Fallback for variable shift right.
+///
+/// When we know that the relevant bits will not overflow, we can instead shift left all
+/// values to align them with the one with the largest right shifts followed by a constant
+/// shift on all values.
+/// In doing so, we replace the variable left shift by a variable multiply with a power of
+/// two.
+///
+/// This trick is borrowed from Daniel Lemire and Leonid Boytsov, Decoding billions of
+/// integers per second through vectorization, Software Practice & Experience 45 (1),
+/// 2015. http://arxiv.org/abs/1209.2137
+template <typename Arch, typename Int, Int... kShifts>
+auto right_shift_by_excess(const xsimd::batch<Int, Arch>& batch,
Review Comment:
It does not work in all cases but works for us. I think this may never be part of `xsimd`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3920825198
Interesting, it looks like the logs got truncated here: https://github.com/apache/arrow/actions/runs/22101636399/job/63889191049?pr=47994#step:12:9485
This has happened before in the case of segfaults. I know that the dataset being used contains nested columns which might provide a bit of a clue. I'll keep looking at the logs to see if there's any other clues beyond truncation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3920652307
@github-actions crossbow submit -g r
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921033582
Thanks for your patience. Conbench analyzed the 2 benchmarking runs that have been run so far on PR commit b63857072c2e9da4cf179cde0479150690ee1932.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/64012120985) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48277: [C++][Parquet] unpack with shuffle algorithm [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #47994:
URL: https://github.com/apache/arrow/pull/47994#issuecomment-3921136157
(deleted the unexpected conbench spam)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org