You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "emkornfield (via GitHub)" <gi...@apache.org> on 2026/05/05 08:30:25 UTC
Re: [PR] GH-48701: [C++][Parquet] Add ALPpd encoding [arrow]
emkornfield commented on code in PR #48345:
URL: https://github.com/apache/arrow/pull/48345#discussion_r3186563721
##########
cpp/cmake_modules/SetupCxxFlags.cmake:
##########
@@ -127,7 +127,7 @@ elseif(ARROW_CPU_FLAG STREQUAL "ppc")
endif()
elseif(ARROW_CPU_FLAG STREQUAL "aarch64")
# Arm64 compiler flags, gcc/clang only
- set(ARROW_ARMV8_MARCH "armv8-a")
+ set(ARROW_ARMV8_MARCH "native")
Review Comment:
I'm not clear on why this change?
##########
cpp/cmake_modules/SetupCxxFlags.cmake:
##########
@@ -621,11 +621,11 @@ endif()
if(NOT MSVC)
set(C_RELEASE_FLAGS "")
if(CMAKE_C_FLAGS_RELEASE MATCHES "-O3")
- string(APPEND C_RELEASE_FLAGS " -O2")
Review Comment:
it seems these are down-graded in more cases below, I think we should probably leave changing these flags to a separate PR so someone with more knowledge on why O2 is used by default can chime in (and we can better track changes here).
##########
cpp/src/arrow/util/alp/alp.h:
##########
@@ -0,0 +1,844 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Adaptive Lossless floating-Point (ALP) compression implementation
+
+#pragma once
+
+#include <optional>
+#include <vector>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// ALP Overview
+//
+// IMPORTANT: For abstract interfaces or examples how to use ALP, consult
+// alp_wrapper.h.
+// This is our implementation of the adaptive lossless floating-point
+// compression for decimals (ALP) (https://dl.acm.org/doi/10.1145/3626717).
+// It works by converting a float into a decimal (if possible). The exponent
+// and factor are chosen per vector. Each float is converted using
+// c(f) = int64(f * 10^exponent * 10^-factor). The converted floats are then
+// encoded via a delta frame of reference and bitpacked. Every exception,
+// where the conversion/reconversion changes the value of the float, is stored
+// separately and has to be patched into the decompressed vector afterwards.
+//
+// ==========================================================================
+// ALP COMPRESSION/DECOMPRESSION PIPELINE
+// ==========================================================================
+//
+// COMPRESSION FLOW:
+// -----------------
+//
+// Input: float/double array
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. SAMPLING & PRESET GENERATION |
+// | * Sample vectors from dataset |
+// | * Try all exponent/factor combinations (e, f) |
+// | * Select best k combinations for preset |
+// +------------------------------------+-----------------------------+
+// | preset.combinations
+// v
+// +------------------------------------------------------------------+
+// | 2. PER-VECTOR COMPRESSION |
+// | a) Find best (e,f) from preset for this vector |
+// | b) Encode: encoded[i] = int64(value[i] * 10^e * 10^-f) |
+// | c) Verify: if decode(encoded[i]) != value[i] -> exception |
+// | d) Replace exceptions with placeholder value |
+// +------------------------------------+-----------------------------+
+// | encoded integers + exceptions
+// v
+// +------------------------------------------------------------------+
+// | 3. FRAME OF REFERENCE (FOR) |
+// | * Find min value in encoded integers |
+// | * Subtract min from all values: delta[i] = encoded[i] - min |
+// +------------------------------------+-----------------------------+
+// | delta values (smaller range)
+// v
+// +------------------------------------------------------------------+
+// | 4. BIT PACKING |
+// | * Calculate bit_width = log2(max_delta) |
+// | * Pack each value into bit_width bits |
+// | * Result: tightly packed binary data |
+// +------------------------------------+-----------------------------+
+// | packed bytes
+// v
+// +------------------------------------------------------------------+
+// | 5. SERIALIZATION (offset-based interleaved layout) |
+// | [Header][Offsets...][Vector₀][Vector₁]... |
+// | where each Vector = [AlpInfo|ForInfo|Data] |
+// +------------------------------------------------------------------+
+//
+//
+// DECOMPRESSION FLOW:
+// -------------------
+//
+// Serialized bytes -> AlpEncodedVector::Load()
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. BIT UNPACKING |
+// | * Extract bit_width from metadata |
+// | * Unpack each value from bit_width bits -> delta values |
+// +------------------------------------+-----------------------------+
+// | delta values
+// v
+// +------------------------------------------------------------------+
+// | 2. REVERSE FRAME OF REFERENCE (unFOR) |
+// | * Add back min: encoded[i] = delta[i] + frame_of_reference |
+// +------------------------------------+-----------------------------+
+// | encoded integers
+// v
+// +------------------------------------------------------------------+
+// | 3. DECODE |
+// | * Apply inverse formula: value[i] = encoded[i] * 10^-e * 10^f |
+// +------------------------------------+-----------------------------+
+// | decoded floats (with placeholders)
+// v
+// +------------------------------------------------------------------+
+// | 4. PATCH EXCEPTIONS |
+// | * Replace values at exception_positions[] with exceptions[] |
+// +------------------------------------+-----------------------------+
+// |
+// v
+// Output: Original float/double array (lossless!)
+//
+// ==========================================================================
+
+// ----------------------------------------------------------------------
+// AlpMode
+
+/// \brief ALP compression mode
+///
+/// Currently only ALP (decimal compression) is implemented.
+enum class AlpMode { kAlp };
+
+// ----------------------------------------------------------------------
+// AlpExponentAndFactor
+
+/// \brief Helper struct to encapsulate the exponent and factor
+struct AlpExponentAndFactor {
+ uint8_t exponent{0};
+ uint8_t factor{0};
+
+ bool operator==(const AlpExponentAndFactor& other) const {
+ return exponent == other.exponent && factor == other.factor;
+ }
+
+ /// \brief Comparison operator for deterministic std::map ordering
+ bool operator<(const AlpExponentAndFactor& other) const {
+ if (exponent != other.exponent) return exponent < other.exponent;
+ return factor < other.factor;
+ }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo (non-templated, ALP core metadata)
+
+/// \brief ALP-specific metadata for an encoded vector (non-templated)
+///
+/// Contains the metadata specific to ALP's float-to-integer conversion:
+/// - exponent/factor: parameters for decimal encoding
+/// - num_exceptions: count of values that couldn't be losslessly encoded
+///
+/// This struct is the same size regardless of the floating-point type (float/double).
+/// It is separate from the integer encoding metadata (e.g., FOR) to allow
+/// different integer encodings to be used in the future.
+///
+/// Serialization format (4 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedVectorInfo (4 bytes) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | exponent (uint8_t) | 1 byte |
+/// | 1 | factor (uint8_t) | 1 byte |
+/// | 2 | num_exceptions | 2 bytes |
+/// +------------------------------------------+
+struct AlpEncodedVectorInfo {
+ /// Exponent used for decimal encoding (multiply by 10^exponent)
+ uint8_t exponent = 0;
+ /// Factor used for decimal encoding (divide by 10^factor)
+ uint8_t factor = 0;
+ /// Number of exceptions stored in this vector
+ uint16_t num_exceptions = 0;
+
+ /// Size of the serialized portion (4 bytes, fixed)
+ static constexpr uint64_t kStoredSize = 4;
Review Comment:
nit: use sizeof(exponent) + sizeof(factor) + size(num_exceptions) to make this clear (and then assert that equal to 4.
##########
cpp/src/arrow/util/alp/alp.cc:
##########
@@ -0,0 +1,1039 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp.h"
+
+#include <cmath>
+#include <cstring>
+#include <functional>
+#include <iostream>
+#include <map>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/bit_stream_utils_internal.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/bpacking_internal.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (frame_of_reference,
+// num_exceptions, offsets) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo implementation (non-templated, 4 bytes)
+
+void AlpEncodedVectorInfo::Store(arrow::util::span<char> output_buffer) const {
+ ARROW_CHECK(output_buffer.size() >= GetStoredSize())
+ << "alp_vector_info_output_too_small: " << output_buffer.size() << " vs "
+ << GetStoredSize();
+
+ char* ptr = output_buffer.data();
+
+ // exponent, factor: 1 byte each
+ *ptr++ = static_cast<char>(exponent);
+ *ptr++ = static_cast<char>(factor);
+
+ // num_exceptions: 2 bytes
+ std::memcpy(ptr, &num_exceptions, sizeof(num_exceptions));
+}
+
+AlpEncodedVectorInfo AlpEncodedVectorInfo::Load(
+ arrow::util::span<const char> input_buffer) {
+ ARROW_CHECK(input_buffer.size() >= GetStoredSize())
+ << "alp_vector_info_input_too_small: " << input_buffer.size() << " vs "
+ << GetStoredSize();
+
+ AlpEncodedVectorInfo result{};
+ const char* ptr = input_buffer.data();
+
+ // exponent, factor: 1 byte each
+ result.exponent = static_cast<uint8_t>(*ptr++);
+ result.factor = static_cast<uint8_t>(*ptr++);
+
+ // num_exceptions: 2 bytes
+ std::memcpy(&result.num_exceptions, ptr, sizeof(result.num_exceptions));
Review Comment:
I think there is a safe store instruction as well.
##########
cpp/src/arrow/util/alp/alp.h:
##########
@@ -0,0 +1,844 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Adaptive Lossless floating-Point (ALP) compression implementation
+
+#pragma once
+
+#include <optional>
+#include <vector>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// ALP Overview
+//
+// IMPORTANT: For abstract interfaces or examples how to use ALP, consult
+// alp_wrapper.h.
+// This is our implementation of the adaptive lossless floating-point
+// compression for decimals (ALP) (https://dl.acm.org/doi/10.1145/3626717).
+// It works by converting a float into a decimal (if possible). The exponent
+// and factor are chosen per vector. Each float is converted using
+// c(f) = int64(f * 10^exponent * 10^-factor). The converted floats are then
+// encoded via a delta frame of reference and bitpacked. Every exception,
+// where the conversion/reconversion changes the value of the float, is stored
+// separately and has to be patched into the decompressed vector afterwards.
+//
+// ==========================================================================
+// ALP COMPRESSION/DECOMPRESSION PIPELINE
+// ==========================================================================
+//
+// COMPRESSION FLOW:
+// -----------------
+//
+// Input: float/double array
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. SAMPLING & PRESET GENERATION |
+// | * Sample vectors from dataset |
+// | * Try all exponent/factor combinations (e, f) |
+// | * Select best k combinations for preset |
+// +------------------------------------+-----------------------------+
+// | preset.combinations
+// v
+// +------------------------------------------------------------------+
+// | 2. PER-VECTOR COMPRESSION |
+// | a) Find best (e,f) from preset for this vector |
+// | b) Encode: encoded[i] = int64(value[i] * 10^e * 10^-f) |
+// | c) Verify: if decode(encoded[i]) != value[i] -> exception |
+// | d) Replace exceptions with placeholder value |
+// +------------------------------------+-----------------------------+
+// | encoded integers + exceptions
+// v
+// +------------------------------------------------------------------+
+// | 3. FRAME OF REFERENCE (FOR) |
+// | * Find min value in encoded integers |
+// | * Subtract min from all values: delta[i] = encoded[i] - min |
+// +------------------------------------+-----------------------------+
+// | delta values (smaller range)
+// v
+// +------------------------------------------------------------------+
+// | 4. BIT PACKING |
+// | * Calculate bit_width = log2(max_delta) |
+// | * Pack each value into bit_width bits |
+// | * Result: tightly packed binary data |
+// +------------------------------------+-----------------------------+
+// | packed bytes
+// v
+// +------------------------------------------------------------------+
+// | 5. SERIALIZATION (offset-based interleaved layout) |
+// | [Header][Offsets...][Vector₀][Vector₁]... |
+// | where each Vector = [AlpInfo|ForInfo|Data] |
+// +------------------------------------------------------------------+
+//
+//
+// DECOMPRESSION FLOW:
+// -------------------
+//
+// Serialized bytes -> AlpEncodedVector::Load()
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. BIT UNPACKING |
+// | * Extract bit_width from metadata |
+// | * Unpack each value from bit_width bits -> delta values |
+// +------------------------------------+-----------------------------+
+// | delta values
+// v
+// +------------------------------------------------------------------+
+// | 2. REVERSE FRAME OF REFERENCE (unFOR) |
+// | * Add back min: encoded[i] = delta[i] + frame_of_reference |
+// +------------------------------------+-----------------------------+
+// | encoded integers
+// v
+// +------------------------------------------------------------------+
+// | 3. DECODE |
+// | * Apply inverse formula: value[i] = encoded[i] * 10^-e * 10^f |
+// +------------------------------------+-----------------------------+
+// | decoded floats (with placeholders)
+// v
+// +------------------------------------------------------------------+
+// | 4. PATCH EXCEPTIONS |
+// | * Replace values at exception_positions[] with exceptions[] |
+// +------------------------------------+-----------------------------+
+// |
+// v
+// Output: Original float/double array (lossless!)
+//
+// ==========================================================================
+
+// ----------------------------------------------------------------------
+// AlpMode
+
+/// \brief ALP compression mode
+///
+/// Currently only ALP (decimal compression) is implemented.
+enum class AlpMode { kAlp };
+
+// ----------------------------------------------------------------------
+// AlpExponentAndFactor
+
+/// \brief Helper struct to encapsulate the exponent and factor
+struct AlpExponentAndFactor {
+ uint8_t exponent{0};
+ uint8_t factor{0};
+
+ bool operator==(const AlpExponentAndFactor& other) const {
+ return exponent == other.exponent && factor == other.factor;
+ }
+
+ /// \brief Comparison operator for deterministic std::map ordering
+ bool operator<(const AlpExponentAndFactor& other) const {
+ if (exponent != other.exponent) return exponent < other.exponent;
+ return factor < other.factor;
+ }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo (non-templated, ALP core metadata)
+
+/// \brief ALP-specific metadata for an encoded vector (non-templated)
+///
+/// Contains the metadata specific to ALP's float-to-integer conversion:
+/// - exponent/factor: parameters for decimal encoding
+/// - num_exceptions: count of values that couldn't be losslessly encoded
+///
+/// This struct is the same size regardless of the floating-point type (float/double).
+/// It is separate from the integer encoding metadata (e.g., FOR) to allow
+/// different integer encodings to be used in the future.
+///
+/// Serialization format (4 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedVectorInfo (4 bytes) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | exponent (uint8_t) | 1 byte |
+/// | 1 | factor (uint8_t) | 1 byte |
+/// | 2 | num_exceptions | 2 bytes |
+/// +------------------------------------------+
+struct AlpEncodedVectorInfo {
+ /// Exponent used for decimal encoding (multiply by 10^exponent)
+ uint8_t exponent = 0;
+ /// Factor used for decimal encoding (divide by 10^factor)
+ uint8_t factor = 0;
+ /// Number of exceptions stored in this vector
+ uint16_t num_exceptions = 0;
+
+ /// Size of the serialized portion (4 bytes, fixed)
+ static constexpr uint64_t kStoredSize = 4;
+
+ /// \brief Store the ALP metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load ALP metadata from an input buffer
+ static AlpEncodedVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the ALP metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get exponent and factor as a combined struct
+ AlpExponentAndFactor GetExponentAndFactor() const {
+ return AlpExponentAndFactor{exponent, factor};
+ }
+
+ bool operator==(const AlpEncodedVectorInfo& other) const {
+ return exponent == other.exponent && factor == other.factor &&
+ num_exceptions == other.num_exceptions;
+ }
+
+ bool operator!=(const AlpEncodedVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedForVectorInfo (templated, FOR integer encoding metadata)
+
+/// \brief FOR (Frame of Reference) encoding metadata for an encoded vector
+///
+/// Contains the metadata specific to FOR bit-packing integer encoding:
+/// - frame_of_reference: minimum value subtracted from all encoded integers
+/// - bit_width: number of bits used to pack each delta value
+///
+/// This struct is templated because frame_of_reference size depends on T:
+/// - float: uint32_t frame_of_reference (4 bytes)
+/// - double: uint64_t frame_of_reference (8 bytes)
+///
+/// Serialization format for float (5 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<float> (5B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 4 bytes |
+/// | 4 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// Serialization format for double (9 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<double> (9B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 8 bytes |
+/// | 8 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+struct AlpEncodedForVectorInfo {
+ static_assert(std::is_same_v<T, float> || std::is_same_v<T, double>,
+ "AlpEncodedForVectorInfo only supports float and double");
+
+ /// Use uint32_t for float, uint64_t for double (matches encoded integer size)
+ using ExactType = typename AlpTypedConstants<T>::FloatingToExact;
+
+ /// Delta used for frame of reference encoding (4 bytes for float, 8 for double)
+ ExactType frame_of_reference = 0;
+ /// Bitwidth used for bitpacking
+ uint8_t bit_width = 0;
+
+ /// Size of the serialized portion (5 bytes for float, 9 for double)
+ static constexpr uint64_t kStoredSize = sizeof(ExactType) + 1;
+
+ /// \brief Compute the bitpacked size in bytes from num_elements and bit_width
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] bit_width bits per element
+ /// \return the size in bytes of the bitpacked data
+ static uint64_t GetBitPackedSize(uint16_t num_elements, uint8_t bit_width) {
+ return (static_cast<uint64_t>(num_elements) * bit_width + 7) / 8;
+ }
+
+ /// \brief Store the FOR metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load FOR metadata from an input buffer
+ static AlpEncodedForVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the FOR metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get the size of the data section (packed values + exceptions)
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] num_exceptions number of exceptions (from AlpEncodedVectorInfo)
+ /// \return the size in bytes of packed values + exception positions + exceptions
+ uint64_t GetDataStoredSize(uint16_t num_elements, uint16_t num_exceptions) const {
+ const uint64_t bit_packed_size = GetBitPackedSize(num_elements, bit_width);
+ return bit_packed_size +
+ num_exceptions * (sizeof(AlpConstants::PositionType) + sizeof(T));
+ }
+
+ bool operator==(const AlpEncodedForVectorInfo& other) const {
+ return frame_of_reference == other.frame_of_reference &&
+ bit_width == other.bit_width;
+ }
+
+ bool operator!=(const AlpEncodedForVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVector
+
+/// \class AlpEncodedVector
+/// \brief A compressed ALP vector with metadata
+///
+/// Per-vector data layout:
+///
+/// +------------------------------------------------------------+
+/// | AlpEncodedVector<T> Data Layout |
+/// +------------------------------------------------------------+
+/// | Section | Size (bytes) | Description |
+/// +-----------------------+----------------------+-------------+
+/// | 1. AlpInfo | 4B (fixed) | ALP meta |
+/// +-----------------------+----------------------+-------------+
+/// | 2. ForInfo | 6B (float) or | FOR meta |
+/// | | 10B (double) | |
+/// +-----------------------+----------------------+-------------+
+/// | 3. Packed Values | bit_packed_size | Bitpacked |
+/// | (compressed data) | (computed) | integers |
+/// +-----------------------+----------------------+-------------+
+/// | 4. Exception Pos | num_exceptions * 2 | uint16_t[] |
+/// | (indices) | (variable) | positions |
+/// +-----------------------+----------------------+-------------+
+/// | 5. Exception Values | num_exceptions * | T[] (float/|
+/// | (original floats) | sizeof(T) | double) |
+/// +------------------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +------------------------------------------------------------+
+/// | Page Layout |
+/// +------------------------------------------------------------+
+/// | [Header (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved |
+/// +------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// The offset-based layout enables:
+/// - O(1) random access to any vector via offset lookup
+/// - Better locality for single-vector decompression
+/// - Parallel decompression without coordination
+///
+/// Example for 1024 floats with 5 exceptions and bit_width=8:
+/// - AlpInfo: 4 bytes (fixed)
+/// - ForInfo: 6 bytes (float)
+/// - Packed Values: 1024 bytes (1024 * 8 bits / 8)
+/// - Exception Pos: 10 bytes (5 * 2)
+/// - Exception Values: 20 bytes (5 * 4)
+/// Total: 1064 bytes
+template <typename T>
+class AlpEncodedVector {
+ public:
+ /// ALP-specific metadata (exponent, factor, num_exceptions)
+ AlpEncodedVectorInfo alp_info;
+ /// FOR-specific metadata (frame_of_reference, bit_width)
+ AlpEncodedForVectorInfo<T> for_info;
+ /// Number of elements in this vector (not serialized; from page header)
+ uint16_t num_elements = 0;
+ /// Successfully encoded and bitpacked data
+ arrow::internal::StaticVector<uint8_t, AlpConstants::kAlpVectorSize * sizeof(T)>
+ packed_values;
+ /// Float values that could not be converted successfully
+ arrow::internal::StaticVector<T, AlpConstants::kAlpVectorSize> exceptions;
+ /// Positions of the exceptions in the decompressed vector
+ arrow::internal::StaticVector<uint16_t, AlpConstants::kAlpVectorSize> exception_positions;
+
+ /// Total metadata size (AlpInfo + ForInfo)
+ static constexpr uint64_t kMetadataStoredSize =
+ AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize;
+
+ /// \brief Get the size of the vector if stored into a sequential memory block
+ ///
+ /// \return the stored size in bytes
+ uint64_t GetStoredSize() const;
+
+ /// \brief Get the stored size for given metadata and element count
+ ///
+ /// \param[in] alp_info the ALP metadata
+ /// \param[in] for_info the FOR metadata
+ /// \param[in] num_elements the number of elements in this vector
+ /// \return the stored size in bytes
+ static uint64_t GetStoredSize(const AlpEncodedVectorInfo& alp_info,
Review Comment:
nit: int64_t
##########
cpp/src/arrow/util/alp/alp.h:
##########
@@ -0,0 +1,844 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Adaptive Lossless floating-Point (ALP) compression implementation
+
+#pragma once
+
+#include <optional>
+#include <vector>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// ALP Overview
+//
+// IMPORTANT: For abstract interfaces or examples how to use ALP, consult
+// alp_wrapper.h.
+// This is our implementation of the adaptive lossless floating-point
+// compression for decimals (ALP) (https://dl.acm.org/doi/10.1145/3626717).
+// It works by converting a float into a decimal (if possible). The exponent
+// and factor are chosen per vector. Each float is converted using
+// c(f) = int64(f * 10^exponent * 10^-factor). The converted floats are then
+// encoded via a delta frame of reference and bitpacked. Every exception,
+// where the conversion/reconversion changes the value of the float, is stored
+// separately and has to be patched into the decompressed vector afterwards.
+//
+// ==========================================================================
+// ALP COMPRESSION/DECOMPRESSION PIPELINE
+// ==========================================================================
+//
+// COMPRESSION FLOW:
+// -----------------
+//
+// Input: float/double array
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. SAMPLING & PRESET GENERATION |
+// | * Sample vectors from dataset |
+// | * Try all exponent/factor combinations (e, f) |
+// | * Select best k combinations for preset |
+// +------------------------------------+-----------------------------+
+// | preset.combinations
+// v
+// +------------------------------------------------------------------+
+// | 2. PER-VECTOR COMPRESSION |
+// | a) Find best (e,f) from preset for this vector |
+// | b) Encode: encoded[i] = int64(value[i] * 10^e * 10^-f) |
+// | c) Verify: if decode(encoded[i]) != value[i] -> exception |
+// | d) Replace exceptions with placeholder value |
+// +------------------------------------+-----------------------------+
+// | encoded integers + exceptions
+// v
+// +------------------------------------------------------------------+
+// | 3. FRAME OF REFERENCE (FOR) |
+// | * Find min value in encoded integers |
+// | * Subtract min from all values: delta[i] = encoded[i] - min |
+// +------------------------------------+-----------------------------+
+// | delta values (smaller range)
+// v
+// +------------------------------------------------------------------+
+// | 4. BIT PACKING |
+// | * Calculate bit_width = log2(max_delta) |
+// | * Pack each value into bit_width bits |
+// | * Result: tightly packed binary data |
+// +------------------------------------+-----------------------------+
+// | packed bytes
+// v
+// +------------------------------------------------------------------+
+// | 5. SERIALIZATION (offset-based interleaved layout) |
+// | [Header][Offsets...][Vector₀][Vector₁]... |
+// | where each Vector = [AlpInfo|ForInfo|Data] |
+// +------------------------------------------------------------------+
+//
+//
+// DECOMPRESSION FLOW:
+// -------------------
+//
+// Serialized bytes -> AlpEncodedVector::Load()
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. BIT UNPACKING |
+// | * Extract bit_width from metadata |
+// | * Unpack each value from bit_width bits -> delta values |
+// +------------------------------------+-----------------------------+
+// | delta values
+// v
+// +------------------------------------------------------------------+
+// | 2. REVERSE FRAME OF REFERENCE (unFOR) |
+// | * Add back min: encoded[i] = delta[i] + frame_of_reference |
+// +------------------------------------+-----------------------------+
+// | encoded integers
+// v
+// +------------------------------------------------------------------+
+// | 3. DECODE |
+// | * Apply inverse formula: value[i] = encoded[i] * 10^-e * 10^f |
+// +------------------------------------+-----------------------------+
+// | decoded floats (with placeholders)
+// v
+// +------------------------------------------------------------------+
+// | 4. PATCH EXCEPTIONS |
+// | * Replace values at exception_positions[] with exceptions[] |
+// +------------------------------------+-----------------------------+
+// |
+// v
+// Output: Original float/double array (lossless!)
+//
+// ==========================================================================
+
+// ----------------------------------------------------------------------
+// AlpMode
+
+/// \brief ALP compression mode
+///
+/// Currently only ALP (decimal compression) is implemented.
+enum class AlpMode { kAlp };
+
+// ----------------------------------------------------------------------
+// AlpExponentAndFactor
+
+/// \brief Helper struct to encapsulate the exponent and factor
+struct AlpExponentAndFactor {
+ uint8_t exponent{0};
+ uint8_t factor{0};
+
+ bool operator==(const AlpExponentAndFactor& other) const {
+ return exponent == other.exponent && factor == other.factor;
+ }
+
+ /// \brief Comparison operator for deterministic std::map ordering
+ bool operator<(const AlpExponentAndFactor& other) const {
+ if (exponent != other.exponent) return exponent < other.exponent;
+ return factor < other.factor;
+ }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo (non-templated, ALP core metadata)
+
+/// \brief ALP-specific metadata for an encoded vector (non-templated)
+///
+/// Contains the metadata specific to ALP's float-to-integer conversion:
+/// - exponent/factor: parameters for decimal encoding
+/// - num_exceptions: count of values that couldn't be losslessly encoded
+///
+/// This struct is the same size regardless of the floating-point type (float/double).
+/// It is separate from the integer encoding metadata (e.g., FOR) to allow
+/// different integer encodings to be used in the future.
+///
+/// Serialization format (4 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedVectorInfo (4 bytes) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | exponent (uint8_t) | 1 byte |
+/// | 1 | factor (uint8_t) | 1 byte |
+/// | 2 | num_exceptions | 2 bytes |
+/// +------------------------------------------+
+struct AlpEncodedVectorInfo {
+ /// Exponent used for decimal encoding (multiply by 10^exponent)
+ uint8_t exponent = 0;
+ /// Factor used for decimal encoding (divide by 10^factor)
+ uint8_t factor = 0;
+ /// Number of exceptions stored in this vector
+ uint16_t num_exceptions = 0;
+
+ /// Size of the serialized portion (4 bytes, fixed)
+ static constexpr uint64_t kStoredSize = 4;
+
+ /// \brief Store the ALP metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load ALP metadata from an input buffer
+ static AlpEncodedVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the ALP metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
Review Comment:
nit: int64_t (or int32_t)?
##########
cpp/src/arrow/util/alp/alp.h:
##########
@@ -0,0 +1,844 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Adaptive Lossless floating-Point (ALP) compression implementation
+
+#pragma once
+
+#include <optional>
+#include <vector>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// ALP Overview
+//
+// IMPORTANT: For abstract interfaces or examples how to use ALP, consult
+// alp_wrapper.h.
+// This is our implementation of the adaptive lossless floating-point
+// compression for decimals (ALP) (https://dl.acm.org/doi/10.1145/3626717).
+// It works by converting a float into a decimal (if possible). The exponent
+// and factor are chosen per vector. Each float is converted using
+// c(f) = int64(f * 10^exponent * 10^-factor). The converted floats are then
+// encoded via a delta frame of reference and bitpacked. Every exception,
+// where the conversion/reconversion changes the value of the float, is stored
+// separately and has to be patched into the decompressed vector afterwards.
+//
+// ==========================================================================
+// ALP COMPRESSION/DECOMPRESSION PIPELINE
+// ==========================================================================
+//
+// COMPRESSION FLOW:
+// -----------------
+//
+// Input: float/double array
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. SAMPLING & PRESET GENERATION |
+// | * Sample vectors from dataset |
+// | * Try all exponent/factor combinations (e, f) |
+// | * Select best k combinations for preset |
+// +------------------------------------+-----------------------------+
+// | preset.combinations
+// v
+// +------------------------------------------------------------------+
+// | 2. PER-VECTOR COMPRESSION |
+// | a) Find best (e,f) from preset for this vector |
+// | b) Encode: encoded[i] = int64(value[i] * 10^e * 10^-f) |
+// | c) Verify: if decode(encoded[i]) != value[i] -> exception |
+// | d) Replace exceptions with placeholder value |
+// +------------------------------------+-----------------------------+
+// | encoded integers + exceptions
+// v
+// +------------------------------------------------------------------+
+// | 3. FRAME OF REFERENCE (FOR) |
+// | * Find min value in encoded integers |
+// | * Subtract min from all values: delta[i] = encoded[i] - min |
+// +------------------------------------+-----------------------------+
+// | delta values (smaller range)
+// v
+// +------------------------------------------------------------------+
+// | 4. BIT PACKING |
+// | * Calculate bit_width = log2(max_delta) |
+// | * Pack each value into bit_width bits |
+// | * Result: tightly packed binary data |
+// +------------------------------------+-----------------------------+
+// | packed bytes
+// v
+// +------------------------------------------------------------------+
+// | 5. SERIALIZATION (offset-based interleaved layout) |
+// | [Header][Offsets...][Vector₀][Vector₁]... |
+// | where each Vector = [AlpInfo|ForInfo|Data] |
+// +------------------------------------------------------------------+
+//
+//
+// DECOMPRESSION FLOW:
+// -------------------
+//
+// Serialized bytes -> AlpEncodedVector::Load()
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. BIT UNPACKING |
+// | * Extract bit_width from metadata |
+// | * Unpack each value from bit_width bits -> delta values |
+// +------------------------------------+-----------------------------+
+// | delta values
+// v
+// +------------------------------------------------------------------+
+// | 2. REVERSE FRAME OF REFERENCE (unFOR) |
+// | * Add back min: encoded[i] = delta[i] + frame_of_reference |
+// +------------------------------------+-----------------------------+
+// | encoded integers
+// v
+// +------------------------------------------------------------------+
+// | 3. DECODE |
+// | * Apply inverse formula: value[i] = encoded[i] * 10^-e * 10^f |
+// +------------------------------------+-----------------------------+
+// | decoded floats (with placeholders)
+// v
+// +------------------------------------------------------------------+
+// | 4. PATCH EXCEPTIONS |
+// | * Replace values at exception_positions[] with exceptions[] |
+// +------------------------------------+-----------------------------+
+// |
+// v
+// Output: Original float/double array (lossless!)
+//
+// ==========================================================================
+
+// ----------------------------------------------------------------------
+// AlpMode
+
+/// \brief ALP compression mode
+///
+/// Currently only ALP (decimal compression) is implemented.
+enum class AlpMode { kAlp };
+
+// ----------------------------------------------------------------------
+// AlpExponentAndFactor
+
+/// \brief Helper struct to encapsulate the exponent and factor
+struct AlpExponentAndFactor {
+ uint8_t exponent{0};
+ uint8_t factor{0};
+
+ bool operator==(const AlpExponentAndFactor& other) const {
+ return exponent == other.exponent && factor == other.factor;
+ }
+
+ /// \brief Comparison operator for deterministic std::map ordering
+ bool operator<(const AlpExponentAndFactor& other) const {
+ if (exponent != other.exponent) return exponent < other.exponent;
+ return factor < other.factor;
+ }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo (non-templated, ALP core metadata)
+
+/// \brief ALP-specific metadata for an encoded vector (non-templated)
+///
+/// Contains the metadata specific to ALP's float-to-integer conversion:
+/// - exponent/factor: parameters for decimal encoding
+/// - num_exceptions: count of values that couldn't be losslessly encoded
+///
+/// This struct is the same size regardless of the floating-point type (float/double).
+/// It is separate from the integer encoding metadata (e.g., FOR) to allow
+/// different integer encodings to be used in the future.
+///
+/// Serialization format (4 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedVectorInfo (4 bytes) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | exponent (uint8_t) | 1 byte |
+/// | 1 | factor (uint8_t) | 1 byte |
+/// | 2 | num_exceptions | 2 bytes |
+/// +------------------------------------------+
+struct AlpEncodedVectorInfo {
+ /// Exponent used for decimal encoding (multiply by 10^exponent)
+ uint8_t exponent = 0;
+ /// Factor used for decimal encoding (divide by 10^factor)
+ uint8_t factor = 0;
+ /// Number of exceptions stored in this vector
+ uint16_t num_exceptions = 0;
+
+ /// Size of the serialized portion (4 bytes, fixed)
+ static constexpr uint64_t kStoredSize = 4;
+
+ /// \brief Store the ALP metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load ALP metadata from an input buffer
+ static AlpEncodedVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the ALP metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get exponent and factor as a combined struct
+ AlpExponentAndFactor GetExponentAndFactor() const {
+ return AlpExponentAndFactor{exponent, factor};
+ }
+
+ bool operator==(const AlpEncodedVectorInfo& other) const {
+ return exponent == other.exponent && factor == other.factor &&
+ num_exceptions == other.num_exceptions;
+ }
+
+ bool operator!=(const AlpEncodedVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedForVectorInfo (templated, FOR integer encoding metadata)
+
+/// \brief FOR (Frame of Reference) encoding metadata for an encoded vector
+///
+/// Contains the metadata specific to FOR bit-packing integer encoding:
+/// - frame_of_reference: minimum value subtracted from all encoded integers
+/// - bit_width: number of bits used to pack each delta value
+///
+/// This struct is templated because frame_of_reference size depends on T:
+/// - float: uint32_t frame_of_reference (4 bytes)
+/// - double: uint64_t frame_of_reference (8 bytes)
+///
+/// Serialization format for float (5 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<float> (5B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 4 bytes |
+/// | 4 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// Serialization format for double (9 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<double> (9B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 8 bytes |
+/// | 8 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+struct AlpEncodedForVectorInfo {
+ static_assert(std::is_same_v<T, float> || std::is_same_v<T, double>,
+ "AlpEncodedForVectorInfo only supports float and double");
+
+ /// Use uint32_t for float, uint64_t for double (matches encoded integer size)
+ using ExactType = typename AlpTypedConstants<T>::FloatingToExact;
+
+ /// Delta used for frame of reference encoding (4 bytes for float, 8 for double)
+ ExactType frame_of_reference = 0;
+ /// Bitwidth used for bitpacking
+ uint8_t bit_width = 0;
+
+ /// Size of the serialized portion (5 bytes for float, 9 for double)
+ static constexpr uint64_t kStoredSize = sizeof(ExactType) + 1;
+
+ /// \brief Compute the bitpacked size in bytes from num_elements and bit_width
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] bit_width bits per element
+ /// \return the size in bytes of the bitpacked data
+ static uint64_t GetBitPackedSize(uint16_t num_elements, uint8_t bit_width) {
+ return (static_cast<uint64_t>(num_elements) * bit_width + 7) / 8;
+ }
+
+ /// \brief Store the FOR metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load FOR metadata from an input buffer
+ static AlpEncodedForVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the FOR metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get the size of the data section (packed values + exceptions)
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] num_exceptions number of exceptions (from AlpEncodedVectorInfo)
+ /// \return the size in bytes of packed values + exception positions + exceptions
+ uint64_t GetDataStoredSize(uint16_t num_elements, uint16_t num_exceptions) const {
+ const uint64_t bit_packed_size = GetBitPackedSize(num_elements, bit_width);
+ return bit_packed_size +
+ num_exceptions * (sizeof(AlpConstants::PositionType) + sizeof(T));
+ }
+
+ bool operator==(const AlpEncodedForVectorInfo& other) const {
+ return frame_of_reference == other.frame_of_reference &&
+ bit_width == other.bit_width;
+ }
+
+ bool operator!=(const AlpEncodedForVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVector
+
+/// \class AlpEncodedVector
+/// \brief A compressed ALP vector with metadata
+///
+/// Per-vector data layout:
+///
+/// +------------------------------------------------------------+
+/// | AlpEncodedVector<T> Data Layout |
+/// +------------------------------------------------------------+
+/// | Section | Size (bytes) | Description |
+/// +-----------------------+----------------------+-------------+
+/// | 1. AlpInfo | 4B (fixed) | ALP meta |
+/// +-----------------------+----------------------+-------------+
+/// | 2. ForInfo | 6B (float) or | FOR meta |
+/// | | 10B (double) | |
+/// +-----------------------+----------------------+-------------+
+/// | 3. Packed Values | bit_packed_size | Bitpacked |
+/// | (compressed data) | (computed) | integers |
+/// +-----------------------+----------------------+-------------+
+/// | 4. Exception Pos | num_exceptions * 2 | uint16_t[] |
+/// | (indices) | (variable) | positions |
+/// +-----------------------+----------------------+-------------+
+/// | 5. Exception Values | num_exceptions * | T[] (float/|
+/// | (original floats) | sizeof(T) | double) |
+/// +------------------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +------------------------------------------------------------+
+/// | Page Layout |
+/// +------------------------------------------------------------+
+/// | [Header (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved |
+/// +------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// The offset-based layout enables:
+/// - O(1) random access to any vector via offset lookup
+/// - Better locality for single-vector decompression
+/// - Parallel decompression without coordination
+///
+/// Example for 1024 floats with 5 exceptions and bit_width=8:
+/// - AlpInfo: 4 bytes (fixed)
+/// - ForInfo: 6 bytes (float)
+/// - Packed Values: 1024 bytes (1024 * 8 bits / 8)
+/// - Exception Pos: 10 bytes (5 * 2)
+/// - Exception Values: 20 bytes (5 * 4)
+/// Total: 1064 bytes
+template <typename T>
+class AlpEncodedVector {
+ public:
+ /// ALP-specific metadata (exponent, factor, num_exceptions)
+ AlpEncodedVectorInfo alp_info;
+ /// FOR-specific metadata (frame_of_reference, bit_width)
+ AlpEncodedForVectorInfo<T> for_info;
+ /// Number of elements in this vector (not serialized; from page header)
+ uint16_t num_elements = 0;
+ /// Successfully encoded and bitpacked data
+ arrow::internal::StaticVector<uint8_t, AlpConstants::kAlpVectorSize * sizeof(T)>
+ packed_values;
+ /// Float values that could not be converted successfully
+ arrow::internal::StaticVector<T, AlpConstants::kAlpVectorSize> exceptions;
+ /// Positions of the exceptions in the decompressed vector
+ arrow::internal::StaticVector<uint16_t, AlpConstants::kAlpVectorSize> exception_positions;
+
+ /// Total metadata size (AlpInfo + ForInfo)
+ static constexpr uint64_t kMetadataStoredSize =
+ AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize;
+
+ /// \brief Get the size of the vector if stored into a sequential memory block
+ ///
+ /// \return the stored size in bytes
+ uint64_t GetStoredSize() const;
+
+ /// \brief Get the stored size for given metadata and element count
+ ///
+ /// \param[in] alp_info the ALP metadata
+ /// \param[in] for_info the FOR metadata
+ /// \param[in] num_elements the number of elements in this vector
+ /// \return the stored size in bytes
+ static uint64_t GetStoredSize(const AlpEncodedVectorInfo& alp_info,
+ const AlpEncodedForVectorInfo<T>& for_info,
+ uint16_t num_elements);
+
+ /// \brief Get the number of elements in this vector
+ ///
+ /// \return number of elements
+ uint64_t GetNumElements() const { return num_elements; }
+
+ /// \brief Store the compressed vector in a compact format into an output buffer
+ ///
+ /// Stores [AlpInfo][ForInfo][PackedValues][ExceptionPositions][ExceptionValues]
+ ///
+ /// \param[out] output_buffer the buffer to store the compressed data into
+ void Store(arrow::util::span<char> output_buffer) const;
Review Comment:
nit: I think we usually use uint8_t to represent bytes in the code base.
##########
cpp/src/arrow/util/alp/alp.h:
##########
@@ -0,0 +1,844 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Adaptive Lossless floating-Point (ALP) compression implementation
+
+#pragma once
+
+#include <optional>
+#include <vector>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// ALP Overview
+//
+// IMPORTANT: For abstract interfaces or examples how to use ALP, consult
+// alp_wrapper.h.
+// This is our implementation of the adaptive lossless floating-point
+// compression for decimals (ALP) (https://dl.acm.org/doi/10.1145/3626717).
+// It works by converting a float into a decimal (if possible). The exponent
+// and factor are chosen per vector. Each float is converted using
+// c(f) = int64(f * 10^exponent * 10^-factor). The converted floats are then
+// encoded via a delta frame of reference and bitpacked. Every exception,
+// where the conversion/reconversion changes the value of the float, is stored
+// separately and has to be patched into the decompressed vector afterwards.
+//
+// ==========================================================================
+// ALP COMPRESSION/DECOMPRESSION PIPELINE
+// ==========================================================================
+//
+// COMPRESSION FLOW:
+// -----------------
+//
+// Input: float/double array
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. SAMPLING & PRESET GENERATION |
+// | * Sample vectors from dataset |
+// | * Try all exponent/factor combinations (e, f) |
+// | * Select best k combinations for preset |
+// +------------------------------------+-----------------------------+
+// | preset.combinations
+// v
+// +------------------------------------------------------------------+
+// | 2. PER-VECTOR COMPRESSION |
+// | a) Find best (e,f) from preset for this vector |
+// | b) Encode: encoded[i] = int64(value[i] * 10^e * 10^-f) |
+// | c) Verify: if decode(encoded[i]) != value[i] -> exception |
+// | d) Replace exceptions with placeholder value |
+// +------------------------------------+-----------------------------+
+// | encoded integers + exceptions
+// v
+// +------------------------------------------------------------------+
+// | 3. FRAME OF REFERENCE (FOR) |
+// | * Find min value in encoded integers |
+// | * Subtract min from all values: delta[i] = encoded[i] - min |
+// +------------------------------------+-----------------------------+
+// | delta values (smaller range)
+// v
+// +------------------------------------------------------------------+
+// | 4. BIT PACKING |
+// | * Calculate bit_width = log2(max_delta) |
+// | * Pack each value into bit_width bits |
+// | * Result: tightly packed binary data |
+// +------------------------------------+-----------------------------+
+// | packed bytes
+// v
+// +------------------------------------------------------------------+
+// | 5. SERIALIZATION (offset-based interleaved layout) |
+// | [Header][Offsets...][Vector₀][Vector₁]... |
+// | where each Vector = [AlpInfo|ForInfo|Data] |
+// +------------------------------------------------------------------+
+//
+//
+// DECOMPRESSION FLOW:
+// -------------------
+//
+// Serialized bytes -> AlpEncodedVector::Load()
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. BIT UNPACKING |
+// | * Extract bit_width from metadata |
+// | * Unpack each value from bit_width bits -> delta values |
+// +------------------------------------+-----------------------------+
+// | delta values
+// v
+// +------------------------------------------------------------------+
+// | 2. REVERSE FRAME OF REFERENCE (unFOR) |
+// | * Add back min: encoded[i] = delta[i] + frame_of_reference |
+// +------------------------------------+-----------------------------+
+// | encoded integers
+// v
+// +------------------------------------------------------------------+
+// | 3. DECODE |
+// | * Apply inverse formula: value[i] = encoded[i] * 10^-e * 10^f |
+// +------------------------------------+-----------------------------+
+// | decoded floats (with placeholders)
+// v
+// +------------------------------------------------------------------+
+// | 4. PATCH EXCEPTIONS |
+// | * Replace values at exception_positions[] with exceptions[] |
+// +------------------------------------+-----------------------------+
+// |
+// v
+// Output: Original float/double array (lossless!)
+//
+// ==========================================================================
+
+// ----------------------------------------------------------------------
+// AlpMode
+
+/// \brief ALP compression mode
+///
+/// Currently only ALP (decimal compression) is implemented.
+enum class AlpMode { kAlp };
+
+// ----------------------------------------------------------------------
+// AlpExponentAndFactor
+
+/// \brief Helper struct to encapsulate the exponent and factor
+struct AlpExponentAndFactor {
+ uint8_t exponent{0};
+ uint8_t factor{0};
+
+ bool operator==(const AlpExponentAndFactor& other) const {
+ return exponent == other.exponent && factor == other.factor;
+ }
+
+ /// \brief Comparison operator for deterministic std::map ordering
+ bool operator<(const AlpExponentAndFactor& other) const {
+ if (exponent != other.exponent) return exponent < other.exponent;
+ return factor < other.factor;
+ }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo (non-templated, ALP core metadata)
+
+/// \brief ALP-specific metadata for an encoded vector (non-templated)
+///
+/// Contains the metadata specific to ALP's float-to-integer conversion:
+/// - exponent/factor: parameters for decimal encoding
+/// - num_exceptions: count of values that couldn't be losslessly encoded
+///
+/// This struct is the same size regardless of the floating-point type (float/double).
+/// It is separate from the integer encoding metadata (e.g., FOR) to allow
+/// different integer encodings to be used in the future.
+///
+/// Serialization format (4 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedVectorInfo (4 bytes) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | exponent (uint8_t) | 1 byte |
+/// | 1 | factor (uint8_t) | 1 byte |
+/// | 2 | num_exceptions | 2 bytes |
+/// +------------------------------------------+
+struct AlpEncodedVectorInfo {
+ /// Exponent used for decimal encoding (multiply by 10^exponent)
+ uint8_t exponent = 0;
+ /// Factor used for decimal encoding (divide by 10^factor)
+ uint8_t factor = 0;
+ /// Number of exceptions stored in this vector
+ uint16_t num_exceptions = 0;
+
+ /// Size of the serialized portion (4 bytes, fixed)
+ static constexpr uint64_t kStoredSize = 4;
+
+ /// \brief Store the ALP metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load ALP metadata from an input buffer
+ static AlpEncodedVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the ALP metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get exponent and factor as a combined struct
+ AlpExponentAndFactor GetExponentAndFactor() const {
+ return AlpExponentAndFactor{exponent, factor};
+ }
+
+ bool operator==(const AlpEncodedVectorInfo& other) const {
+ return exponent == other.exponent && factor == other.factor &&
+ num_exceptions == other.num_exceptions;
+ }
+
+ bool operator!=(const AlpEncodedVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedForVectorInfo (templated, FOR integer encoding metadata)
+
+/// \brief FOR (Frame of Reference) encoding metadata for an encoded vector
+///
+/// Contains the metadata specific to FOR bit-packing integer encoding:
+/// - frame_of_reference: minimum value subtracted from all encoded integers
+/// - bit_width: number of bits used to pack each delta value
+///
+/// This struct is templated because frame_of_reference size depends on T:
+/// - float: uint32_t frame_of_reference (4 bytes)
+/// - double: uint64_t frame_of_reference (8 bytes)
+///
+/// Serialization format for float (5 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<float> (5B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 4 bytes |
+/// | 4 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// Serialization format for double (9 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<double> (9B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 8 bytes |
+/// | 8 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+struct AlpEncodedForVectorInfo {
+ static_assert(std::is_same_v<T, float> || std::is_same_v<T, double>,
+ "AlpEncodedForVectorInfo only supports float and double");
+
+ /// Use uint32_t for float, uint64_t for double (matches encoded integer size)
+ using ExactType = typename AlpTypedConstants<T>::FloatingToExact;
+
+ /// Delta used for frame of reference encoding (4 bytes for float, 8 for double)
+ ExactType frame_of_reference = 0;
+ /// Bitwidth used for bitpacking
+ uint8_t bit_width = 0;
+
+ /// Size of the serialized portion (5 bytes for float, 9 for double)
+ static constexpr uint64_t kStoredSize = sizeof(ExactType) + 1;
+
+ /// \brief Compute the bitpacked size in bytes from num_elements and bit_width
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] bit_width bits per element
+ /// \return the size in bytes of the bitpacked data
+ static uint64_t GetBitPackedSize(uint16_t num_elements, uint8_t bit_width) {
+ return (static_cast<uint64_t>(num_elements) * bit_width + 7) / 8;
+ }
+
+ /// \brief Store the FOR metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load FOR metadata from an input buffer
+ static AlpEncodedForVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the FOR metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get the size of the data section (packed values + exceptions)
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] num_exceptions number of exceptions (from AlpEncodedVectorInfo)
+ /// \return the size in bytes of packed values + exception positions + exceptions
+ uint64_t GetDataStoredSize(uint16_t num_elements, uint16_t num_exceptions) const {
+ const uint64_t bit_packed_size = GetBitPackedSize(num_elements, bit_width);
+ return bit_packed_size +
+ num_exceptions * (sizeof(AlpConstants::PositionType) + sizeof(T));
+ }
+
+ bool operator==(const AlpEncodedForVectorInfo& other) const {
+ return frame_of_reference == other.frame_of_reference &&
+ bit_width == other.bit_width;
+ }
+
+ bool operator!=(const AlpEncodedForVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVector
+
+/// \class AlpEncodedVector
+/// \brief A compressed ALP vector with metadata
+///
+/// Per-vector data layout:
+///
+/// +------------------------------------------------------------+
+/// | AlpEncodedVector<T> Data Layout |
+/// +------------------------------------------------------------+
+/// | Section | Size (bytes) | Description |
+/// +-----------------------+----------------------+-------------+
+/// | 1. AlpInfo | 4B (fixed) | ALP meta |
+/// +-----------------------+----------------------+-------------+
+/// | 2. ForInfo | 6B (float) or | FOR meta |
+/// | | 10B (double) | |
+/// +-----------------------+----------------------+-------------+
+/// | 3. Packed Values | bit_packed_size | Bitpacked |
+/// | (compressed data) | (computed) | integers |
+/// +-----------------------+----------------------+-------------+
+/// | 4. Exception Pos | num_exceptions * 2 | uint16_t[] |
+/// | (indices) | (variable) | positions |
+/// +-----------------------+----------------------+-------------+
+/// | 5. Exception Values | num_exceptions * | T[] (float/|
+/// | (original floats) | sizeof(T) | double) |
+/// +------------------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +------------------------------------------------------------+
+/// | Page Layout |
+/// +------------------------------------------------------------+
+/// | [Header (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved |
Review Comment:
```suggestion
/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Concatenated |
```
##########
cpp/src/arrow/util/alp/alp_wrapper.h:
##########
@@ -0,0 +1,185 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// High-level wrapper interface for ALP compression
+
+#pragma once
+
+#include <cstddef>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_sampler.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// AlpCodec
+
+/// \class AlpCodec
+/// \brief High-level interface for ALP compression
+///
+/// AlpCodec is an interface for Adaptive Lossless floating-Point Compression
+/// (ALP) (https://dl.acm.org/doi/10.1145/3626717). For encoding, it samples
+/// the data and applies decimal compression (Alp) to floating point values.
+/// This class acts as a wrapper around the vector-based interfaces of
+/// AlpSampler and Alp.
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+class AlpCodec {
+ public:
+ /// Type alias for the sampler result containing encoding presets
+ using AlpSamplerResult = typename AlpSampler<T>::AlpSamplerResult;
+
+ /// \brief Create a sampling preset from input data
+ ///
+ /// This samples the input data and generates an encoding preset that can be
+ /// reused for encoding. This is useful when you want to pre-compute the preset
+ /// outside of the benchmark loop or encode multiple batches with the same preset.
+ ///
+ /// \param[in] decomp pointer to the input data to sample
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \return the sampling result containing the encoding preset
+ static AlpSamplerResult CreateSamplingPreset(const T* decomp, size_t decomp_size);
+
+ /// \brief Encode floating point values using a pre-computed preset
+ ///
+ /// This encodes the data using a preset that was previously computed via
+ /// CreateSamplingPreset(). This avoids the sampling overhead during encoding.
+ ///
+ /// \param[in] decomp pointer to the input that is to be encoded
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \param[out] comp pointer to the memory region we will encode into.
+ /// Must be at least GetMaxCompressedSize(decomp_size) bytes.
+ /// \param[in,out] comp_size the actual size of the encoded data in bytes,
+ /// expects the size of comp as input. If this is too small,
+ /// this is set to 0 and we bail out.
+ /// \param[in] preset the pre-computed sampling result from CreateSamplingPreset()
+ static void EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset);
+
+ /// \brief Encode floating point values using ALP decimal compression
+ ///
+ /// \param[in] decomp pointer to the input that is to be encoded
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \param[out] comp pointer to the memory region we will encode into.
+ /// Must be at least GetMaxCompressedSize(decomp_size) bytes.
+ /// \param[in,out] comp_size the actual size of the encoded data in bytes,
+ /// expects the size of comp as input. If this is too small,
+ /// this is set to 0 and we bail out.
+ /// \param[in] enforce_mode reserved for future use.
+ /// Currently only AlpMode::kAlp is supported.
+ static void Encode(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size,
+ std::optional<AlpMode> enforce_mode = std::nullopt);
+
+ /// \brief Decode floating point values
+ ///
+ /// \param[in] num_elements number of elements to decode (from page header)
+ /// \param[in] comp pointer to the input that is to be decoded
+ /// \param[in] comp_size size of the input in bytes (from page header)
+ /// \param[out] decomp pointer to the memory region we will decode into.
+ /// The caller is responsible for ensuring this is big enough
+ /// to hold num_elements values.
+ /// \return Status::OK on success, or an error if the compressed data is malformed
+ /// \tparam TargetType the type that is used to store the output.
+ /// May not be a narrowing conversion from T.
+ template <typename TargetType>
+ static Status Decode(int32_t num_elements, const char* comp, size_t comp_size,
+ TargetType* decomp);
+
+ /// \brief Get the maximum compressed size of an uncompressed buffer
+ ///
+ /// \param[in] uncompressed_size the size of the uncompressed buffer in bytes
+ /// \return the maximum size of the compressed buffer
+ static int64_t GetMaxCompressedSize(int64_t uncompressed_size);
+
+ private:
+ struct AlpHeader;
+
+ /// \brief Tracks the progress of a compression operation
+ ///
+ /// Used to report how much data was consumed and produced during encoding.
+ struct CompressionProgress {
+ /// Number of compressed bytes written to output
+ int64_t num_compressed_bytes_produced = 0;
+ /// Number of input elements consumed
+ int64_t num_uncompressed_elements_taken = 0;
+ };
+
+ /// \brief Tracks the progress of a decompression operation
+ ///
+ /// Used to report how much data was consumed and produced during decoding.
+ struct DecompressionProgress {
+ /// Number of decompressed elements written
+ int64_t num_decompressed_elements_produced = 0;
+ /// Number of compressed bytes consumed
+ int64_t num_compressed_bytes_taken = 0;
+ };
+
+ /// \brief Compress a buffer using the ALP variant
+ ///
+ /// \param[in] decomp array of floating point numbers to compress
+ /// \param[in] element_count the number of floating point numbers
+ /// \param[out] comp the buffer to be compressed into
+ /// \param[in] comp_size the size of the compression buffer
+ /// \param[in] combinations the encoding preset to use
+ /// \return the compression progress
+ static CompressionProgress EncodeAlp(const T* decomp, uint64_t element_count,
Review Comment:
same naming bikeshed, but reverse (`decomp`->`input`, comp->'buffer')
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
+ header.num_elements = static_cast<uint32_t>(element_count);
Review Comment:
should be int32_t I think?
##########
cpp/src/arrow/util/alp/alp_wrapper.h:
##########
@@ -0,0 +1,185 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// High-level wrapper interface for ALP compression
+
+#pragma once
+
+#include <cstddef>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_sampler.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// AlpCodec
+
+/// \class AlpCodec
+/// \brief High-level interface for ALP compression
+///
+/// AlpCodec is an interface for Adaptive Lossless floating-Point Compression
+/// (ALP) (https://dl.acm.org/doi/10.1145/3626717). For encoding, it samples
+/// the data and applies decimal compression (Alp) to floating point values.
+/// This class acts as a wrapper around the vector-based interfaces of
+/// AlpSampler and Alp.
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+class AlpCodec {
+ public:
+ /// Type alias for the sampler result containing encoding presets
+ using AlpSamplerResult = typename AlpSampler<T>::AlpSamplerResult;
+
+ /// \brief Create a sampling preset from input data
+ ///
+ /// This samples the input data and generates an encoding preset that can be
+ /// reused for encoding. This is useful when you want to pre-compute the preset
+ /// outside of the benchmark loop or encode multiple batches with the same preset.
+ ///
+ /// \param[in] decomp pointer to the input data to sample
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \return the sampling result containing the encoding preset
+ static AlpSamplerResult CreateSamplingPreset(const T* decomp, size_t decomp_size);
+
+ /// \brief Encode floating point values using a pre-computed preset
+ ///
+ /// This encodes the data using a preset that was previously computed via
+ /// CreateSamplingPreset(). This avoids the sampling overhead during encoding.
+ ///
+ /// \param[in] decomp pointer to the input that is to be encoded
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \param[out] comp pointer to the memory region we will encode into.
+ /// Must be at least GetMaxCompressedSize(decomp_size) bytes.
+ /// \param[in,out] comp_size the actual size of the encoded data in bytes,
+ /// expects the size of comp as input. If this is too small,
+ /// this is set to 0 and we bail out.
+ /// \param[in] preset the pre-computed sampling result from CreateSamplingPreset()
+ static void EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset);
+
+ /// \brief Encode floating point values using ALP decimal compression
+ ///
+ /// \param[in] decomp pointer to the input that is to be encoded
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \param[out] comp pointer to the memory region we will encode into.
+ /// Must be at least GetMaxCompressedSize(decomp_size) bytes.
+ /// \param[in,out] comp_size the actual size of the encoded data in bytes,
+ /// expects the size of comp as input. If this is too small,
+ /// this is set to 0 and we bail out.
+ /// \param[in] enforce_mode reserved for future use.
+ /// Currently only AlpMode::kAlp is supported.
+ static void Encode(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size,
+ std::optional<AlpMode> enforce_mode = std::nullopt);
+
+ /// \brief Decode floating point values
+ ///
+ /// \param[in] num_elements number of elements to decode (from page header)
+ /// \param[in] comp pointer to the input that is to be decoded
+ /// \param[in] comp_size size of the input in bytes (from page header)
+ /// \param[out] decomp pointer to the memory region we will decode into.
+ /// The caller is responsible for ensuring this is big enough
+ /// to hold num_elements values.
+ /// \return Status::OK on success, or an error if the compressed data is malformed
+ /// \tparam TargetType the type that is used to store the output.
+ /// May not be a narrowing conversion from T.
+ template <typename TargetType>
+ static Status Decode(int32_t num_elements, const char* comp, size_t comp_size,
+ TargetType* decomp);
+
+ /// \brief Get the maximum compressed size of an uncompressed buffer
+ ///
+ /// \param[in] uncompressed_size the size of the uncompressed buffer in bytes
+ /// \return the maximum size of the compressed buffer
+ static int64_t GetMaxCompressedSize(int64_t uncompressed_size);
+
+ private:
+ struct AlpHeader;
+
+ /// \brief Tracks the progress of a compression operation
+ ///
+ /// Used to report how much data was consumed and produced during encoding.
+ struct CompressionProgress {
+ /// Number of compressed bytes written to output
+ int64_t num_compressed_bytes_produced = 0;
+ /// Number of input elements consumed
+ int64_t num_uncompressed_elements_taken = 0;
+ };
+
+ /// \brief Tracks the progress of a decompression operation
+ ///
+ /// Used to report how much data was consumed and produced during decoding.
+ struct DecompressionProgress {
+ /// Number of decompressed elements written
+ int64_t num_decompressed_elements_produced = 0;
+ /// Number of compressed bytes consumed
+ int64_t num_compressed_bytes_taken = 0;
+ };
+
+ /// \brief Compress a buffer using the ALP variant
+ ///
+ /// \param[in] decomp array of floating point numbers to compress
+ /// \param[in] element_count the number of floating point numbers
+ /// \param[out] comp the buffer to be compressed into
+ /// \param[in] comp_size the size of the compression buffer
+ /// \param[in] combinations the encoding preset to use
+ /// \return the compression progress
+ static CompressionProgress EncodeAlp(const T* decomp, uint64_t element_count,
+ char* comp, size_t comp_size,
+ const AlpEncodingParameters& combinations);
+
+ /// \brief Decompress a buffer using the ALP variant
+ ///
+ /// \param[in] decomp_element_count the number of floats to decompress
+ /// \param[in] comp the compressed buffer to be decompressed
+ /// \param[in] comp_size the size of the compressed data
+ /// \param[in] integer_encoding the bit packing layout used
+ /// \param[in] vector_size the number of elements per vector (from header)
+ /// \param[in] total_elements the total number of elements in the page (from header).
+ /// Uses uint32_t since Parquet page headers use i32 for num_values.
+ /// \param[out] decomp the buffer to be decompressed into
+ /// \return the decompression progress, or an error if the compressed data is malformed
+ /// \tparam TargetType the type that is used to store the output.
+ /// May not be a narrowing conversion from T.
+ template <typename TargetType>
+ static Result<DecompressionProgress> DecodeAlp(size_t decomp_element_count,
+ const char* comp, size_t comp_size,
+ AlpIntegerEncoding integer_encoding,
+ uint32_t vector_size,
Review Comment:
signed ints?
##########
cpp/src/arrow/util/alp/alp_wrapper.h:
##########
@@ -0,0 +1,185 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// High-level wrapper interface for ALP compression
+
+#pragma once
+
+#include <cstddef>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_sampler.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// AlpCodec
+
+/// \class AlpCodec
+/// \brief High-level interface for ALP compression
+///
+/// AlpCodec is an interface for Adaptive Lossless floating-Point Compression
+/// (ALP) (https://dl.acm.org/doi/10.1145/3626717). For encoding, it samples
+/// the data and applies decimal compression (Alp) to floating point values.
+/// This class acts as a wrapper around the vector-based interfaces of
+/// AlpSampler and Alp.
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+class AlpCodec {
+ public:
+ /// Type alias for the sampler result containing encoding presets
+ using AlpSamplerResult = typename AlpSampler<T>::AlpSamplerResult;
+
+ /// \brief Create a sampling preset from input data
+ ///
+ /// This samples the input data and generates an encoding preset that can be
+ /// reused for encoding. This is useful when you want to pre-compute the preset
+ /// outside of the benchmark loop or encode multiple batches with the same preset.
+ ///
+ /// \param[in] decomp pointer to the input data to sample
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \return the sampling result containing the encoding preset
+ static AlpSamplerResult CreateSamplingPreset(const T* decomp, size_t decomp_size);
+
+ /// \brief Encode floating point values using a pre-computed preset
+ ///
+ /// This encodes the data using a preset that was previously computed via
+ /// CreateSamplingPreset(). This avoids the sampling overhead during encoding.
+ ///
+ /// \param[in] decomp pointer to the input that is to be encoded
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \param[out] comp pointer to the memory region we will encode into.
+ /// Must be at least GetMaxCompressedSize(decomp_size) bytes.
+ /// \param[in,out] comp_size the actual size of the encoded data in bytes,
+ /// expects the size of comp as input. If this is too small,
+ /// this is set to 0 and we bail out.
+ /// \param[in] preset the pre-computed sampling result from CreateSamplingPreset()
+ static void EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset);
+
+ /// \brief Encode floating point values using ALP decimal compression
+ ///
+ /// \param[in] decomp pointer to the input that is to be encoded
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \param[out] comp pointer to the memory region we will encode into.
+ /// Must be at least GetMaxCompressedSize(decomp_size) bytes.
+ /// \param[in,out] comp_size the actual size of the encoded data in bytes,
+ /// expects the size of comp as input. If this is too small,
+ /// this is set to 0 and we bail out.
+ /// \param[in] enforce_mode reserved for future use.
+ /// Currently only AlpMode::kAlp is supported.
+ static void Encode(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size,
+ std::optional<AlpMode> enforce_mode = std::nullopt);
+
+ /// \brief Decode floating point values
+ ///
+ /// \param[in] num_elements number of elements to decode (from page header)
+ /// \param[in] comp pointer to the input that is to be decoded
+ /// \param[in] comp_size size of the input in bytes (from page header)
+ /// \param[out] decomp pointer to the memory region we will decode into.
+ /// The caller is responsible for ensuring this is big enough
+ /// to hold num_elements values.
+ /// \return Status::OK on success, or an error if the compressed data is malformed
+ /// \tparam TargetType the type that is used to store the output.
+ /// May not be a narrowing conversion from T.
+ template <typename TargetType>
+ static Status Decode(int32_t num_elements, const char* comp, size_t comp_size,
+ TargetType* decomp);
+
+ /// \brief Get the maximum compressed size of an uncompressed buffer
+ ///
+ /// \param[in] uncompressed_size the size of the uncompressed buffer in bytes
+ /// \return the maximum size of the compressed buffer
+ static int64_t GetMaxCompressedSize(int64_t uncompressed_size);
+
+ private:
+ struct AlpHeader;
+
+ /// \brief Tracks the progress of a compression operation
+ ///
+ /// Used to report how much data was consumed and produced during encoding.
+ struct CompressionProgress {
+ /// Number of compressed bytes written to output
+ int64_t num_compressed_bytes_produced = 0;
+ /// Number of input elements consumed
+ int64_t num_uncompressed_elements_taken = 0;
+ };
+
+ /// \brief Tracks the progress of a decompression operation
+ ///
+ /// Used to report how much data was consumed and produced during decoding.
+ struct DecompressionProgress {
+ /// Number of decompressed elements written
+ int64_t num_decompressed_elements_produced = 0;
+ /// Number of compressed bytes consumed
+ int64_t num_compressed_bytes_taken = 0;
+ };
+
+ /// \brief Compress a buffer using the ALP variant
+ ///
+ /// \param[in] decomp array of floating point numbers to compress
+ /// \param[in] element_count the number of floating point numbers
+ /// \param[out] comp the buffer to be compressed into
+ /// \param[in] comp_size the size of the compression buffer
+ /// \param[in] combinations the encoding preset to use
+ /// \return the compression progress
+ static CompressionProgress EncodeAlp(const T* decomp, uint64_t element_count,
+ char* comp, size_t comp_size,
Review Comment:
nit: comp should go last.
##########
cpp/src/parquet/encoding_benchmark.cc:
##########
@@ -661,6 +661,78 @@ BENCHMARK(BM_ByteStreamSplitEncode_Float_Neon)->Apply(ByteStreamSplitApply);
BENCHMARK(BM_ByteStreamSplitEncode_Double_Neon)->Apply(ByteStreamSplitApply);
#endif
+// ----------------------------------------------------------------------
+// ALP encoding/decoding benchmarks
+
+static void BM_AlpEncodingFloat(benchmark::State& state) {
+ std::vector<float> values(state.range(0), 64.0f);
+ auto encoder = MakeTypedEncoder<FloatType>(Encoding::ALP);
+ for (auto _ : state) {
+ encoder->Put(values.data(), static_cast<int>(values.size()));
+ encoder->FlushValues();
+ }
+ state.SetBytesProcessed(state.iterations() * state.range(0) * sizeof(float));
+ state.SetItemsProcessed(state.iterations() * state.range(0));
+}
+
+BENCHMARK(BM_AlpEncodingFloat)->Range(MIN_RANGE, MAX_RANGE);
+
+static void BM_AlpDecodingFloat(benchmark::State& state) {
+ std::vector<float> values(state.range(0), 64.0f);
+ auto encoder = MakeTypedEncoder<FloatType>(Encoding::ALP);
+ encoder->Put(values.data(), static_cast<int>(values.size()));
+ std::shared_ptr<Buffer> buf = encoder->FlushValues();
+
+ for (auto _ : state) {
+ auto decoder = MakeTypedDecoder<FloatType>(Encoding::ALP);
+ decoder->SetData(static_cast<int>(values.size()), buf->data(),
+ static_cast<int>(buf->size()));
+ std::vector<float> output(values.size());
+ decoder->Decode(output.data(), static_cast<int>(values.size()));
+ benchmark::ClobberMemory();
+ }
+ state.SetBytesProcessed(state.iterations() * state.range(0) * sizeof(float));
+ state.SetItemsProcessed(state.iterations() * state.range(0));
+}
+
+BENCHMARK(BM_AlpDecodingFloat)->Range(MIN_RANGE, MAX_RANGE);
+
+static void BM_AlpEncodingDouble(benchmark::State& state) {
+ std::vector<double> values(state.range(0), 64.0);
+ auto encoder = MakeTypedEncoder<DoubleType>(Encoding::ALP);
+ for (auto _ : state) {
+ encoder->Put(values.data(), static_cast<int>(values.size()));
+ encoder->FlushValues();
+ }
+ state.SetBytesProcessed(state.iterations() * state.range(0) * sizeof(double));
+ state.SetItemsProcessed(state.iterations() * state.range(0));
+}
+
+BENCHMARK(BM_AlpEncodingDouble)->Range(MIN_RANGE, MAX_RANGE);
+
+static void BM_AlpDecodingDouble(benchmark::State& state) {
+ std::vector<double> values(state.range(0), 64.0);
+ auto encoder = MakeTypedEncoder<DoubleType>(Encoding::ALP);
+ encoder->Put(values.data(), static_cast<int>(values.size()));
+ std::shared_ptr<Buffer> buf = encoder->FlushValues();
+
+ for (auto _ : state) {
+ auto decoder = MakeTypedDecoder<DoubleType>(Encoding::ALP);
Review Comment:
nit: move this out of the benchmark loop?
##########
cpp/src/arrow/util/alp/ALP_Encoding_Specification.md:
##########
@@ -0,0 +1,696 @@
+# ALP Encoding Specification
Review Comment:
bump.
##########
cpp/src/arrow/util/alp/alp.h:
##########
@@ -0,0 +1,844 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Adaptive Lossless floating-Point (ALP) compression implementation
+
+#pragma once
+
+#include <optional>
+#include <vector>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// ALP Overview
+//
+// IMPORTANT: For abstract interfaces or examples how to use ALP, consult
+// alp_wrapper.h.
+// This is our implementation of the adaptive lossless floating-point
+// compression for decimals (ALP) (https://dl.acm.org/doi/10.1145/3626717).
+// It works by converting a float into a decimal (if possible). The exponent
+// and factor are chosen per vector. Each float is converted using
+// c(f) = int64(f * 10^exponent * 10^-factor). The converted floats are then
+// encoded via a delta frame of reference and bitpacked. Every exception,
+// where the conversion/reconversion changes the value of the float, is stored
+// separately and has to be patched into the decompressed vector afterwards.
+//
+// ==========================================================================
+// ALP COMPRESSION/DECOMPRESSION PIPELINE
+// ==========================================================================
+//
+// COMPRESSION FLOW:
+// -----------------
+//
+// Input: float/double array
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. SAMPLING & PRESET GENERATION |
+// | * Sample vectors from dataset |
+// | * Try all exponent/factor combinations (e, f) |
+// | * Select best k combinations for preset |
+// +------------------------------------+-----------------------------+
+// | preset.combinations
+// v
+// +------------------------------------------------------------------+
+// | 2. PER-VECTOR COMPRESSION |
+// | a) Find best (e,f) from preset for this vector |
+// | b) Encode: encoded[i] = int64(value[i] * 10^e * 10^-f) |
+// | c) Verify: if decode(encoded[i]) != value[i] -> exception |
+// | d) Replace exceptions with placeholder value |
+// +------------------------------------+-----------------------------+
+// | encoded integers + exceptions
+// v
+// +------------------------------------------------------------------+
+// | 3. FRAME OF REFERENCE (FOR) |
+// | * Find min value in encoded integers |
+// | * Subtract min from all values: delta[i] = encoded[i] - min |
+// +------------------------------------+-----------------------------+
+// | delta values (smaller range)
+// v
+// +------------------------------------------------------------------+
+// | 4. BIT PACKING |
+// | * Calculate bit_width = log2(max_delta) |
+// | * Pack each value into bit_width bits |
+// | * Result: tightly packed binary data |
+// +------------------------------------+-----------------------------+
+// | packed bytes
+// v
+// +------------------------------------------------------------------+
+// | 5. SERIALIZATION (offset-based interleaved layout) |
+// | [Header][Offsets...][Vector₀][Vector₁]... |
+// | where each Vector = [AlpInfo|ForInfo|Data] |
+// +------------------------------------------------------------------+
+//
+//
+// DECOMPRESSION FLOW:
+// -------------------
+//
+// Serialized bytes -> AlpEncodedVector::Load()
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. BIT UNPACKING |
+// | * Extract bit_width from metadata |
+// | * Unpack each value from bit_width bits -> delta values |
+// +------------------------------------+-----------------------------+
+// | delta values
+// v
+// +------------------------------------------------------------------+
+// | 2. REVERSE FRAME OF REFERENCE (unFOR) |
+// | * Add back min: encoded[i] = delta[i] + frame_of_reference |
+// +------------------------------------+-----------------------------+
+// | encoded integers
+// v
+// +------------------------------------------------------------------+
+// | 3. DECODE |
+// | * Apply inverse formula: value[i] = encoded[i] * 10^-e * 10^f |
+// +------------------------------------+-----------------------------+
+// | decoded floats (with placeholders)
+// v
+// +------------------------------------------------------------------+
+// | 4. PATCH EXCEPTIONS |
+// | * Replace values at exception_positions[] with exceptions[] |
+// +------------------------------------+-----------------------------+
+// |
+// v
+// Output: Original float/double array (lossless!)
+//
+// ==========================================================================
+
+// ----------------------------------------------------------------------
+// AlpMode
+
+/// \brief ALP compression mode
+///
+/// Currently only ALP (decimal compression) is implemented.
+enum class AlpMode { kAlp };
+
+// ----------------------------------------------------------------------
+// AlpExponentAndFactor
+
+/// \brief Helper struct to encapsulate the exponent and factor
+struct AlpExponentAndFactor {
+ uint8_t exponent{0};
+ uint8_t factor{0};
+
+ bool operator==(const AlpExponentAndFactor& other) const {
+ return exponent == other.exponent && factor == other.factor;
+ }
+
+ /// \brief Comparison operator for deterministic std::map ordering
+ bool operator<(const AlpExponentAndFactor& other) const {
+ if (exponent != other.exponent) return exponent < other.exponent;
+ return factor < other.factor;
+ }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo (non-templated, ALP core metadata)
+
+/// \brief ALP-specific metadata for an encoded vector (non-templated)
+///
+/// Contains the metadata specific to ALP's float-to-integer conversion:
+/// - exponent/factor: parameters for decimal encoding
+/// - num_exceptions: count of values that couldn't be losslessly encoded
+///
+/// This struct is the same size regardless of the floating-point type (float/double).
+/// It is separate from the integer encoding metadata (e.g., FOR) to allow
+/// different integer encodings to be used in the future.
+///
+/// Serialization format (4 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedVectorInfo (4 bytes) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | exponent (uint8_t) | 1 byte |
+/// | 1 | factor (uint8_t) | 1 byte |
+/// | 2 | num_exceptions | 2 bytes |
+/// +------------------------------------------+
+struct AlpEncodedVectorInfo {
+ /// Exponent used for decimal encoding (multiply by 10^exponent)
+ uint8_t exponent = 0;
+ /// Factor used for decimal encoding (divide by 10^factor)
+ uint8_t factor = 0;
+ /// Number of exceptions stored in this vector
+ uint16_t num_exceptions = 0;
+
+ /// Size of the serialized portion (4 bytes, fixed)
+ static constexpr uint64_t kStoredSize = 4;
+
+ /// \brief Store the ALP metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load ALP metadata from an input buffer
+ static AlpEncodedVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the ALP metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get exponent and factor as a combined struct
+ AlpExponentAndFactor GetExponentAndFactor() const {
+ return AlpExponentAndFactor{exponent, factor};
+ }
+
+ bool operator==(const AlpEncodedVectorInfo& other) const {
+ return exponent == other.exponent && factor == other.factor &&
+ num_exceptions == other.num_exceptions;
+ }
+
+ bool operator!=(const AlpEncodedVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedForVectorInfo (templated, FOR integer encoding metadata)
+
+/// \brief FOR (Frame of Reference) encoding metadata for an encoded vector
+///
+/// Contains the metadata specific to FOR bit-packing integer encoding:
+/// - frame_of_reference: minimum value subtracted from all encoded integers
+/// - bit_width: number of bits used to pack each delta value
+///
+/// This struct is templated because frame_of_reference size depends on T:
+/// - float: uint32_t frame_of_reference (4 bytes)
+/// - double: uint64_t frame_of_reference (8 bytes)
+///
+/// Serialization format for float (5 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<float> (5B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 4 bytes |
+/// | 4 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// Serialization format for double (9 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<double> (9B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 8 bytes |
+/// | 8 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+struct AlpEncodedForVectorInfo {
+ static_assert(std::is_same_v<T, float> || std::is_same_v<T, double>,
+ "AlpEncodedForVectorInfo only supports float and double");
+
+ /// Use uint32_t for float, uint64_t for double (matches encoded integer size)
+ using ExactType = typename AlpTypedConstants<T>::FloatingToExact;
+
+ /// Delta used for frame of reference encoding (4 bytes for float, 8 for double)
+ ExactType frame_of_reference = 0;
+ /// Bitwidth used for bitpacking
+ uint8_t bit_width = 0;
+
+ /// Size of the serialized portion (5 bytes for float, 9 for double)
+ static constexpr uint64_t kStoredSize = sizeof(ExactType) + 1;
+
+ /// \brief Compute the bitpacked size in bytes from num_elements and bit_width
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] bit_width bits per element
+ /// \return the size in bytes of the bitpacked data
+ static uint64_t GetBitPackedSize(uint16_t num_elements, uint8_t bit_width) {
+ return (static_cast<uint64_t>(num_elements) * bit_width + 7) / 8;
+ }
+
+ /// \brief Store the FOR metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load FOR metadata from an input buffer
+ static AlpEncodedForVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the FOR metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get the size of the data section (packed values + exceptions)
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] num_exceptions number of exceptions (from AlpEncodedVectorInfo)
+ /// \return the size in bytes of packed values + exception positions + exceptions
+ uint64_t GetDataStoredSize(uint16_t num_elements, uint16_t num_exceptions) const {
+ const uint64_t bit_packed_size = GetBitPackedSize(num_elements, bit_width);
+ return bit_packed_size +
+ num_exceptions * (sizeof(AlpConstants::PositionType) + sizeof(T));
+ }
+
+ bool operator==(const AlpEncodedForVectorInfo& other) const {
+ return frame_of_reference == other.frame_of_reference &&
+ bit_width == other.bit_width;
+ }
+
+ bool operator!=(const AlpEncodedForVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVector
+
+/// \class AlpEncodedVector
+/// \brief A compressed ALP vector with metadata
+///
+/// Per-vector data layout:
+///
+/// +------------------------------------------------------------+
+/// | AlpEncodedVector<T> Data Layout |
+/// +------------------------------------------------------------+
+/// | Section | Size (bytes) | Description |
+/// +-----------------------+----------------------+-------------+
+/// | 1. AlpInfo | 4B (fixed) | ALP meta |
+/// +-----------------------+----------------------+-------------+
+/// | 2. ForInfo | 6B (float) or | FOR meta |
+/// | | 10B (double) | |
+/// +-----------------------+----------------------+-------------+
+/// | 3. Packed Values | bit_packed_size | Bitpacked |
+/// | (compressed data) | (computed) | integers |
+/// +-----------------------+----------------------+-------------+
+/// | 4. Exception Pos | num_exceptions * 2 | uint16_t[] |
+/// | (indices) | (variable) | positions |
+/// +-----------------------+----------------------+-------------+
+/// | 5. Exception Values | num_exceptions * | T[] (float/|
+/// | (original floats) | sizeof(T) | double) |
+/// +------------------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +------------------------------------------------------------+
+/// | Page Layout |
+/// +------------------------------------------------------------+
+/// | [Header (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved |
+/// +------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// The offset-based layout enables:
+/// - O(1) random access to any vector via offset lookup
+/// - Better locality for single-vector decompression
+/// - Parallel decompression without coordination
+///
+/// Example for 1024 floats with 5 exceptions and bit_width=8:
+/// - AlpInfo: 4 bytes (fixed)
+/// - ForInfo: 6 bytes (float)
+/// - Packed Values: 1024 bytes (1024 * 8 bits / 8)
+/// - Exception Pos: 10 bytes (5 * 2)
+/// - Exception Values: 20 bytes (5 * 4)
+/// Total: 1064 bytes
+template <typename T>
+class AlpEncodedVector {
+ public:
+ /// ALP-specific metadata (exponent, factor, num_exceptions)
+ AlpEncodedVectorInfo alp_info;
+ /// FOR-specific metadata (frame_of_reference, bit_width)
+ AlpEncodedForVectorInfo<T> for_info;
+ /// Number of elements in this vector (not serialized; from page header)
+ uint16_t num_elements = 0;
+ /// Successfully encoded and bitpacked data
+ arrow::internal::StaticVector<uint8_t, AlpConstants::kAlpVectorSize * sizeof(T)>
+ packed_values;
+ /// Float values that could not be converted successfully
+ arrow::internal::StaticVector<T, AlpConstants::kAlpVectorSize> exceptions;
+ /// Positions of the exceptions in the decompressed vector
+ arrow::internal::StaticVector<uint16_t, AlpConstants::kAlpVectorSize> exception_positions;
+
+ /// Total metadata size (AlpInfo + ForInfo)
+ static constexpr uint64_t kMetadataStoredSize =
+ AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize;
+
+ /// \brief Get the size of the vector if stored into a sequential memory block
+ ///
+ /// \return the stored size in bytes
+ uint64_t GetStoredSize() const;
Review Comment:
nit: int64_t
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
+ header.num_elements = static_cast<uint32_t>(element_count);
+
+ std::memcpy(encoded_header, &header.compression_mode, 3);
+ std::memcpy(encoded_header + 3, &header.num_elements, sizeof(header.num_elements));
+ *comp_size = AlpHeader::kSize + compression_progress.num_compressed_bytes_produced;
+}
+
+template <typename T>
+void AlpCodec<T>::Encode(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, std::optional<AlpMode> enforce_mode) {
+ // Sample the data and encode with the preset
+ auto sampling_result = CreateSamplingPreset(decomp, decomp_size);
+ EncodeWithPreset(decomp, decomp_size, comp, comp_size, sampling_result);
+}
+
+template <typename T>
+template <typename TargetType>
+Status AlpCodec<T>::Decode(int32_t num_elements, const char* comp, size_t comp_size,
+ TargetType* decomp) {
+ ARROW_ASSIGN_OR_RAISE(const AlpHeader header, LoadHeader(comp, comp_size));
+ if (header.log_vector_size > AlpConstants::kMaxLogVectorSize) {
+ return Status::Invalid("ALP log_vector_size too large: ",
+ static_cast<int>(header.log_vector_size),
+ " > ", static_cast<int>(AlpConstants::kMaxLogVectorSize),
+ " (would overflow uint16_t element count)");
+ }
+ const uint32_t vector_size = header.GetVectorSize();
+ if (vector_size != AlpConstants::kAlpVectorSize) {
+ return Status::Invalid("Unsupported ALP vector_size: ", vector_size,
+ " (only ", AlpConstants::kAlpVectorSize, " is supported)");
+ }
+
+ const char* compression_body = comp + AlpHeader::kSize;
+ const uint64_t compression_body_size = comp_size - AlpHeader::kSize;
+
+ if (header.GetCompressionMode() != AlpMode::kAlp) {
+ return Status::Invalid("Unsupported ALP compression mode: ",
+ static_cast<int>(header.compression_mode));
+ }
+
+ ARROW_RETURN_NOT_OK(
+ DecodeAlp<TargetType>(num_elements, compression_body, compression_body_size,
+ header.GetIntegerEncoding(), vector_size,
+ header.num_elements, decomp)
+ .status());
+ return Status::OK();
+}
+
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, float* decomp);
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+template Status AlpCodec<double>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+
+template <typename T>
+int64_t AlpCodec<T>::GetMaxCompressedSize(int64_t uncompressed_size) {
+ ARROW_CHECK(uncompressed_size >= 0 && uncompressed_size % sizeof(T) == 0)
+ << "alp_decompressed_size_not_multiple_of_T";
+ const uint64_t element_count = static_cast<uint64_t>(uncompressed_size) / sizeof(T);
+ uint64_t max_alp_size = AlpHeader::kSize;
+
+ const uint64_t vectors_count =
+ static_cast<uint64_t>(std::ceil(static_cast<double>(element_count) / AlpConstants::kAlpVectorSize));
+
+ // Add offsets section (4 bytes per vector)
+ max_alp_size += vectors_count * sizeof(AlpConstants::OffsetType);
+
+ // Add per-vector metadata sizes: AlpInfo (4 bytes) + ForInfo (5/9 bytes)
+ max_alp_size +=
+ (AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize) * vectors_count;
+
+ // Worst case: everything is an exception, except two values that are chosen
+ // with large difference to make FOR encoding for placeholders impossible.
+ // Values/placeholders.
+ max_alp_size += element_count * sizeof(T);
+ // Exceptions.
+ max_alp_size += element_count * sizeof(T);
+ // Exception positions.
+ max_alp_size += element_count * sizeof(AlpConstants::PositionType);
+
+ return static_cast<int64_t>(max_alp_size);
+}
+
+template <typename T>
+auto AlpCodec<T>::EncodeAlp(const T* decomp, uint64_t element_count, char* comp,
+ size_t comp_size, const AlpEncodingParameters& combinations)
+ -> CompressionProgress {
+ // OFFSET-BASED LAYOUT
+ // [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Byte offsets to each vector (4B each)
+ // [AlpInfo₀ | ForInfo₀ | Data₀] ← Vector 0 (interleaved)
+ // [AlpInfo₁ | ForInfo₁ | Data₁] ← Vector 1
+ // ...
+ // [AlpInfoₙ₋₁ | ForInfoₙ₋₁ | Dataₙ₋₁] ← Vector n-1
+ //
+ // Benefits:
+ // - O(1) random access to any vector (no cumulative offset computation)
+ // - Better locality for single-vector access (metadata + data together)
+ // - Enables parallel decompression without coordination
+
+ // Phase 1: Compress all vectors and collect them
+ std::vector<AlpEncodedVector<T>> encoded_vectors;
+ const uint64_t num_vectors =
+ (element_count + AlpConstants::kAlpVectorSize - 1) / AlpConstants::kAlpVectorSize;
+ encoded_vectors.reserve(num_vectors);
+
+ uint64_t input_offset = 0;
+ for (uint64_t remaining_elements = element_count; remaining_elements > 0;
+ remaining_elements -= std::min(AlpConstants::kAlpVectorSize, remaining_elements)) {
+ const uint64_t elements_to_encode =
+ std::min(AlpConstants::kAlpVectorSize, remaining_elements);
+ encoded_vectors.push_back(AlpCompression<T>::CompressVector(
+ decomp + input_offset, static_cast<uint16_t>(elements_to_encode), combinations));
+ input_offset += elements_to_encode;
+ }
+
+ // Phase 2: Calculate sizes and offsets
+ const AlpIntegerEncoding integer_encoding = combinations.integer_encoding;
+ const uint64_t per_vector_metadata_size =
+ AlpEncodedVectorInfo::kStoredSize + GetIntegerEncodingMetadataSize<T>(integer_encoding);
+
+ // Offsets section comes first (after header, which is written by Encode())
+ const uint64_t offsets_section_size =
+ num_vectors * sizeof(AlpConstants::OffsetType);
+
+ // Calculate total size and per-vector offsets
+ std::vector<AlpConstants::OffsetType> vector_offsets;
+ vector_offsets.reserve(num_vectors);
+
+ // First vector starts right after the offsets section
+ uint64_t current_offset = offsets_section_size;
+ for (const auto& vec : encoded_vectors) {
+ // Store offset to this vector (relative to start of body, after header)
+ vector_offsets.push_back(static_cast<AlpConstants::OffsetType>(current_offset));
+ // Advance by metadata + data size
+ current_offset += per_vector_metadata_size + vec.GetDataStoredSize();
+ }
+ const uint64_t total_size = current_offset;
+
+ if (total_size > comp_size) {
+ return CompressionProgress{0, 0};
+ }
+
+ // Phase 3: Write offsets section
+ char* offset_ptr = comp;
+ for (const auto& offset : vector_offsets) {
+ std::memcpy(offset_ptr, &offset, sizeof(AlpConstants::OffsetType));
+ offset_ptr += sizeof(AlpConstants::OffsetType);
+ }
+
+ // Phase 4: Write interleaved vectors [AlpInfo | ForInfo | Data]
+ for (size_t i = 0; i < encoded_vectors.size(); i++) {
+ const auto& vec = encoded_vectors[i];
+ char* vector_start = comp + vector_offsets[i];
+
+ // Write AlpInfo
+ vec.alp_info.Store({vector_start, AlpEncodedVectorInfo::kStoredSize});
+ char* ptr = vector_start + AlpEncodedVectorInfo::kStoredSize;
+
+ // Write ForInfo (or other integer encoding metadata)
+ switch (integer_encoding) {
+ case AlpIntegerEncoding::kForBitPack: {
+ vec.for_info.Store({ptr, AlpEncodedForVectorInfo<T>::kStoredSize});
+ ptr += AlpEncodedForVectorInfo<T>::kStoredSize;
+ } break;
+
+ default:
+ ARROW_CHECK(false) << "unsupported_integer_encoding: "
+ << static_cast<int>(integer_encoding);
+ break;
+ }
+
+ // Write data (packed values + exception positions + exception values)
+ const uint64_t data_size = vec.GetDataStoredSize();
+ vec.StoreDataOnly({ptr, data_size});
+ }
+
+ return CompressionProgress{static_cast<int64_t>(total_size),
+ static_cast<int64_t>(element_count)};
+}
+
+template <typename T>
+template <typename TargetType>
+auto AlpCodec<T>::DecodeAlp(size_t decomp_element_count,
+ const char* comp, size_t comp_size,
+ AlpIntegerEncoding integer_encoding,
+ uint32_t vector_size, uint32_t total_elements,
+ TargetType* decomp)
+ -> Result<DecompressionProgress> {
+ // OFFSET-BASED LAYOUT:
+ // [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Byte offsets to each vector (4B each)
+ // [AlpInfo₀ | ForInfo₀ | Data₀] ← Vector 0 (interleaved)
+ // [AlpInfo₁ | ForInfo₁ | Data₁] ← Vector 1
+ // ...
+ //
+ // Benefits:
+ // - O(1) random access to any vector (no cumulative offset computation)
+ // - Better locality for single-vector access (metadata + data together)
+ // - Enables parallel decompression without coordination
+
+ // Calculate number of vectors
+ const uint32_t num_vectors =
+ (total_elements + vector_size - 1) / vector_size;
+
+ if (num_vectors == 0) {
+ return DecompressionProgress{0, 0};
+ }
+
+ const uint64_t offsets_section_size =
+ static_cast<uint64_t>(num_vectors) * sizeof(AlpConstants::OffsetType);
+ if (comp_size < offsets_section_size) {
+ return Status::Invalid("ALP compressed buffer too small for offsets section: ",
+ comp_size, " < ", offsets_section_size);
+ }
+
+ // Read all offsets
+ std::vector<AlpConstants::OffsetType> vector_offsets(num_vectors);
+ std::memcpy(vector_offsets.data(), comp,
+ num_vectors * sizeof(AlpConstants::OffsetType));
+
+ // Decode each vector using its offset for O(1) random access
+ uint64_t output_offset = 0;
+ uint64_t bytes_consumed = offsets_section_size;
+
+ for (uint32_t vector_index = 0; vector_index < num_vectors; vector_index++) {
+ // Calculate number of elements in this vector
+ const uint64_t num_full_vectors = total_elements / vector_size;
+ const uint64_t remainder = total_elements % vector_size;
+ uint16_t this_vector_elements;
+ if (vector_index < num_full_vectors) {
+ this_vector_elements = static_cast<uint16_t>(vector_size);
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ this_vector_elements = static_cast<uint16_t>(remainder);
+ } else {
+ this_vector_elements = 0;
+ }
+
+ if (output_offset + this_vector_elements > decomp_element_count) {
+ return Status::Invalid("ALP decode output buffer too small: offset=",
+ output_offset, " + elements=", this_vector_elements,
+ " > capacity=", decomp_element_count);
+ }
+
+ // Validate offset is within bounds and enough buffer remains for metadata
+ const uint64_t vector_offset = vector_offsets[vector_index];
+ constexpr uint64_t kMinVectorMetadataSize =
+ AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize;
+ if (vector_offset + kMinVectorMetadataSize > comp_size) {
+ return Status::Invalid("ALP vector offset out of bounds or insufficient buffer "
+ "for metadata: offset=", vector_offset,
+ ", metadata_size=", kMinVectorMetadataSize,
+ ", buffer_size=", comp_size);
+ }
+
+ // Jump directly to this vector using its offset
+ const char* vector_start = comp + vector_offset;
+ const uint64_t remaining = comp_size - vector_offset;
+
+ // Read AlpInfo (interleaved)
+ const AlpEncodedVectorInfo alp_info =
+ AlpEncodedVectorInfo::Load({vector_start, remaining});
+ const char* ptr = vector_start + AlpEncodedVectorInfo::kStoredSize;
+
+ // Decode based on integer encoding type
+ switch (integer_encoding) {
Review Comment:
instead of switch here for now consider refactoring to guards that return early on invalid data (avoids a nesting level).
##########
cpp/src/arrow/util/alp/alp.h:
##########
@@ -0,0 +1,843 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Adaptive Lossless floating-Point (ALP) compression implementation
+
+#pragma once
+
+#include <vector>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// ALP Overview
+//
+// IMPORTANT: For abstract interfaces or examples how to use ALP, consult
+// alp_wrapper.h.
+// This is our implementation of the adaptive lossless floating-point
+// compression for decimals (ALP) (https://dl.acm.org/doi/10.1145/3626717).
+// It works by converting a float into a decimal (if possible). The exponent
+// and factor are chosen per vector. Each float is converted using
+// c(f) = int64(f * 10^exponent * 10^-factor). The converted floats are then
+// encoded via a delta frame of reference and bitpacked. Every exception,
+// where the conversion/reconversion changes the value of the float, is stored
+// separately and has to be patched into the decompressed vector afterwards.
+//
+// ==========================================================================
+// ALP COMPRESSION/DECOMPRESSION PIPELINE
+// ==========================================================================
+//
+// COMPRESSION FLOW:
+// -----------------
+//
+// Input: float/double array
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. SAMPLING & PRESET GENERATION |
+// | * Sample vectors from dataset |
+// | * Try all exponent/factor combinations (e, f) |
+// | * Select best k combinations for preset |
+// +------------------------------------+-----------------------------+
+// | preset.combinations
+// v
+// +------------------------------------------------------------------+
+// | 2. PER-VECTOR COMPRESSION |
+// | a) Find best (e,f) from preset for this vector |
+// | b) Encode: encoded[i] = int64(value[i] * 10^e * 10^-f) |
+// | c) Verify: if decode(encoded[i]) != value[i] -> exception |
+// | d) Replace exceptions with placeholder value |
+// +------------------------------------+-----------------------------+
+// | encoded integers + exceptions
+// v
+// +------------------------------------------------------------------+
+// | 3. FRAME OF REFERENCE (FOR) |
+// | * Find min value in encoded integers |
+// | * Subtract min from all values: delta[i] = encoded[i] - min |
+// +------------------------------------+-----------------------------+
+// | delta values (smaller range)
+// v
+// +------------------------------------------------------------------+
+// | 4. BIT PACKING |
+// | * Calculate bit_width = log2(max_delta) |
+// | * Pack each value into bit_width bits |
+// | * Result: tightly packed binary data |
+// +------------------------------------+-----------------------------+
+// | packed bytes
+// v
+// +------------------------------------------------------------------+
+// | 5. SERIALIZATION (offset-based interleaved layout) |
+// | [Header][Offsets...][Vector₀][Vector₁]... |
+// | where each Vector = [AlpInfo|ForInfo|Data] |
+// +------------------------------------------------------------------+
+//
+//
+// DECOMPRESSION FLOW:
+// -------------------
+//
+// Serialized bytes -> AlpEncodedVector::Load()
+// |
+// v
+// +------------------------------------------------------------------+
+// | 1. BIT UNPACKING |
+// | * Extract bit_width from metadata |
+// | * Unpack each value from bit_width bits -> delta values |
+// +------------------------------------+-----------------------------+
+// | delta values
+// v
+// +------------------------------------------------------------------+
+// | 2. REVERSE FRAME OF REFERENCE (unFOR) |
+// | * Add back min: encoded[i] = delta[i] + frame_of_reference |
+// +------------------------------------+-----------------------------+
+// | encoded integers
+// v
+// +------------------------------------------------------------------+
+// | 3. DECODE |
+// | * Apply inverse formula: value[i] = encoded[i] * 10^-e * 10^f |
+// +------------------------------------+-----------------------------+
+// | decoded floats (with placeholders)
+// v
+// +------------------------------------------------------------------+
+// | 4. PATCH EXCEPTIONS |
+// | * Replace values at exception_positions[] with exceptions[] |
+// +------------------------------------+-----------------------------+
+// |
+// v
+// Output: Original float/double array (lossless!)
+//
+// ==========================================================================
+
+// ----------------------------------------------------------------------
+// AlpMode
+
+/// \brief ALP compression mode
+///
+/// Currently only ALP (decimal compression) is implemented.
+enum class AlpMode { kAlp };
+
+// ----------------------------------------------------------------------
+// AlpExponentAndFactor
+
+/// \brief Helper struct to encapsulate the exponent and factor
+struct AlpExponentAndFactor {
+ uint8_t exponent{0};
+ uint8_t factor{0};
+
+ bool operator==(const AlpExponentAndFactor& other) const {
+ return exponent == other.exponent && factor == other.factor;
+ }
+
+ /// \brief Comparison operator for deterministic std::map ordering
+ bool operator<(const AlpExponentAndFactor& other) const {
+ if (exponent != other.exponent) return exponent < other.exponent;
+ return factor < other.factor;
+ }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo (non-templated, ALP core metadata)
+
+/// \brief ALP-specific metadata for an encoded vector (non-templated)
+///
+/// Contains the metadata specific to ALP's float-to-integer conversion:
+/// - exponent/factor: parameters for decimal encoding
+/// - num_exceptions: count of values that couldn't be losslessly encoded
+///
+/// This struct is the same size regardless of the floating-point type (float/double).
+/// It is separate from the integer encoding metadata (e.g., FOR) to allow
+/// different integer encodings to be used in the future.
+///
+/// Serialization format (4 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedVectorInfo (4 bytes) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | exponent (uint8_t) | 1 byte |
+/// | 1 | factor (uint8_t) | 1 byte |
+/// | 2 | num_exceptions | 2 bytes |
+/// +------------------------------------------+
+struct AlpEncodedVectorInfo {
+ /// Exponent used for decimal encoding (multiply by 10^exponent)
+ uint8_t exponent = 0;
+ /// Factor used for decimal encoding (divide by 10^factor)
+ uint8_t factor = 0;
+ /// Number of exceptions stored in this vector
+ uint16_t num_exceptions = 0;
+
+ /// Size of the serialized portion (4 bytes, fixed)
+ static constexpr uint64_t kStoredSize = 4;
+
+ /// \brief Store the ALP metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load ALP metadata from an input buffer
+ static AlpEncodedVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the ALP metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get exponent and factor as a combined struct
+ AlpExponentAndFactor GetExponentAndFactor() const {
+ return AlpExponentAndFactor{exponent, factor};
+ }
+
+ bool operator==(const AlpEncodedVectorInfo& other) const {
+ return exponent == other.exponent && factor == other.factor &&
+ num_exceptions == other.num_exceptions;
+ }
+
+ bool operator!=(const AlpEncodedVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedForVectorInfo (templated, FOR integer encoding metadata)
+
+/// \brief FOR (Frame of Reference) encoding metadata for an encoded vector
+///
+/// Contains the metadata specific to FOR bit-packing integer encoding:
+/// - frame_of_reference: minimum value subtracted from all encoded integers
+/// - bit_width: number of bits used to pack each delta value
+///
+/// This struct is templated because frame_of_reference size depends on T:
+/// - float: uint32_t frame_of_reference (4 bytes)
+/// - double: uint64_t frame_of_reference (8 bytes)
+///
+/// Serialization format for float (5 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<float> (5B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 4 bytes |
+/// | 4 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// Serialization format for double (9 bytes):
+///
+/// +------------------------------------------+
+/// | AlpEncodedForVectorInfo<double> (9B) |
+/// +------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+----------+
+/// | 0 | frame_of_reference | 8 bytes |
+/// | 8 | bit_width (uint8_t)| 1 byte |
+/// +------------------------------------------+
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+struct AlpEncodedForVectorInfo {
+ static_assert(std::is_same_v<T, float> || std::is_same_v<T, double>,
+ "AlpEncodedForVectorInfo only supports float and double");
+
+ /// Use uint32_t for float, uint64_t for double (matches encoded integer size)
+ using ExactType = typename AlpTypedConstants<T>::FloatingToExact;
+
+ /// Delta used for frame of reference encoding (4 bytes for float, 8 for double)
+ ExactType frame_of_reference = 0;
+ /// Bitwidth used for bitpacking
+ uint8_t bit_width = 0;
+
+ /// Size of the serialized portion (5 bytes for float, 9 for double)
+ static constexpr uint64_t kStoredSize = sizeof(ExactType) + 1;
+
+ /// \brief Compute the bitpacked size in bytes from num_elements and bit_width
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] bit_width bits per element
+ /// \return the size in bytes of the bitpacked data
+ static uint64_t GetBitPackedSize(uint16_t num_elements, uint8_t bit_width) {
+ return (static_cast<uint64_t>(num_elements) * bit_width + 7) / 8;
+ }
+
+ /// \brief Store the FOR metadata into an output buffer
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Load FOR metadata from an input buffer
+ static AlpEncodedForVectorInfo Load(arrow::util::span<const char> input_buffer);
+
+ /// \brief Get serialized size of the FOR metadata
+ static uint64_t GetStoredSize() { return kStoredSize; }
+
+ /// \brief Get the size of the data section (packed values + exceptions)
+ ///
+ /// \param[in] num_elements number of elements in this vector
+ /// \param[in] num_exceptions number of exceptions (from AlpEncodedVectorInfo)
+ /// \return the size in bytes of packed values + exception positions + exceptions
+ uint64_t GetDataStoredSize(uint16_t num_elements, uint16_t num_exceptions) const {
+ const uint64_t bit_packed_size = GetBitPackedSize(num_elements, bit_width);
+ return bit_packed_size +
+ num_exceptions * (sizeof(AlpConstants::PositionType) + sizeof(T));
+ }
+
+ bool operator==(const AlpEncodedForVectorInfo& other) const {
+ return frame_of_reference == other.frame_of_reference &&
+ bit_width == other.bit_width;
+ }
+
+ bool operator!=(const AlpEncodedForVectorInfo& other) const { return !(*this == other); }
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVector
+
+/// \class AlpEncodedVector
+/// \brief A compressed ALP vector with metadata
+///
+/// Per-vector data layout:
+///
+/// +------------------------------------------------------------+
+/// | AlpEncodedVector<T> Data Layout |
+/// +------------------------------------------------------------+
+/// | Section | Size (bytes) | Description |
+/// +-----------------------+----------------------+-------------+
+/// | 1. AlpInfo | 4B (fixed) | ALP meta |
+/// +-----------------------+----------------------+-------------+
+/// | 2. ForInfo | 6B (float) or | FOR meta |
+/// | | 10B (double) | |
+/// +-----------------------+----------------------+-------------+
+/// | 3. Packed Values | bit_packed_size | Bitpacked |
+/// | (compressed data) | (computed) | integers |
+/// +-----------------------+----------------------+-------------+
+/// | 4. Exception Pos | num_exceptions * 2 | uint16_t[] |
+/// | (indices) | (variable) | positions |
+/// +-----------------------+----------------------+-------------+
+/// | 5. Exception Values | num_exceptions * | T[] (float/|
+/// | (original floats) | sizeof(T) | double) |
+/// +------------------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +------------------------------------------------------------+
+/// | Page Layout |
+/// +------------------------------------------------------------+
+/// | [Header (8B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved |
+/// +------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// The offset-based layout enables:
+/// - O(1) random access to any vector via offset lookup
+/// - Better locality for single-vector decompression
+/// - Parallel decompression without coordination
+///
+/// Example for 1024 floats with 5 exceptions and bit_width=8:
+/// - AlpInfo: 4 bytes (fixed)
+/// - ForInfo: 6 bytes (float)
+/// - Packed Values: 1024 bytes (1024 * 8 bits / 8)
+/// - Exception Pos: 10 bytes (5 * 2)
+/// - Exception Values: 20 bytes (5 * 4)
+/// Total: 1064 bytes
+template <typename T>
+class AlpEncodedVector {
+ public:
+ /// ALP-specific metadata (exponent, factor, num_exceptions)
+ AlpEncodedVectorInfo alp_info;
+ /// FOR-specific metadata (frame_of_reference, bit_width)
+ AlpEncodedForVectorInfo<T> for_info;
+ /// Number of elements in this vector (not serialized; from page header)
+ uint16_t num_elements = 0;
+ /// Successfully encoded and bitpacked data
+ arrow::internal::StaticVector<uint8_t, AlpConstants::kAlpVectorSize * sizeof(T)>
+ packed_values;
+ /// Float values that could not be converted successfully
+ arrow::internal::StaticVector<T, AlpConstants::kAlpVectorSize> exceptions;
+ /// Positions of the exceptions in the decompressed vector
+ arrow::internal::StaticVector<uint16_t, AlpConstants::kAlpVectorSize> exception_positions;
+
+ /// Total metadata size (AlpInfo + ForInfo)
+ static constexpr uint64_t kMetadataStoredSize =
+ AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize;
+
+ /// \brief Get the size of the vector if stored into a sequential memory block
+ ///
+ /// \return the stored size in bytes
+ uint64_t GetStoredSize() const;
+
+ /// \brief Get the stored size for given metadata and element count
+ ///
+ /// \param[in] alp_info the ALP metadata
+ /// \param[in] for_info the FOR metadata
+ /// \param[in] num_elements the number of elements in this vector
+ /// \return the stored size in bytes
+ static uint64_t GetStoredSize(const AlpEncodedVectorInfo& alp_info,
+ const AlpEncodedForVectorInfo<T>& for_info,
+ uint16_t num_elements);
+
+ /// \brief Get the number of elements in this vector
+ ///
+ /// \return number of elements
+ uint64_t GetNumElements() const { return num_elements; }
+
+ /// \brief Store the compressed vector in a compact format into an output buffer
+ ///
+ /// Stores [AlpInfo][ForInfo][PackedValues][ExceptionPositions][ExceptionValues]
+ ///
+ /// \param[out] output_buffer the buffer to store the compressed data into
+ void Store(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Store only the data section (without metadata) into an output buffer
+ ///
+ /// Stores [PackedValues][ExceptionPositions][ExceptionValues]
+ /// Used when metadata (AlpInfo, ForInfo) is written separately.
+ ///
+ /// \param[out] output_buffer the buffer to store the data section into
+ void StoreDataOnly(arrow::util::span<char> output_buffer) const;
+
+ /// \brief Get the size of the data section only (without metadata)
+ ///
+ /// \return the size in bytes of packed values + exception positions + exceptions
+ uint64_t GetDataStoredSize() const {
+ return for_info.GetDataStoredSize(num_elements, alp_info.num_exceptions);
+ }
+
+ /// \brief Load a compressed vector from a compact format from an input buffer
+ ///
+ /// \param[in] input_buffer the buffer to load from
+ /// \param[in] num_elements the number of elements (from page header)
+ /// \return the loaded AlpEncodedVector
+ static AlpEncodedVector Load(arrow::util::span<const char> input_buffer,
+ uint16_t num_elements);
+
+ bool operator==(const AlpEncodedVector<T>& other) const;
+};
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorView
+
+/// \class AlpEncodedVectorView
+/// \brief A view into compressed ALP data optimized for decompression
+///
+/// Unlike AlpEncodedVector which copies all data into internal buffers,
+/// AlpEncodedVectorView uses zero-copy for the large packed values array
+/// while copying the small exception arrays into aligned storage.
+///
+/// The packed values are accessed via a span (zero-copy) since they are
+/// byte arrays with no alignment requirements. Exception positions and
+/// values are copied into aligned StaticVectors because:
+/// 1. The serialized data may not be properly aligned for uint16_t/T access
+/// 2. Exceptions are rare (typically < 5%), so copying is negligible
+/// 3. This avoids undefined behavior from misaligned memory access
+///
+/// Use LoadView() to create a view, then pass to DecompressVectorView().
+/// The underlying buffer must remain valid for the lifetime of the view
+/// (for packed_values access).
+template <typename T>
+struct AlpEncodedVectorView {
+ /// ALP-specific metadata (exponent, factor, num_exceptions)
+ AlpEncodedVectorInfo alp_info;
+ /// FOR-specific metadata (frame_of_reference, bit_width)
+ AlpEncodedForVectorInfo<T> for_info;
+ /// Number of elements in this vector (not serialized; from page header)
+ uint16_t num_elements = 0;
Review Comment:
We need to update the spec for this if that is going to be a limitation.
##########
cpp/src/arrow/util/alp/alp.cc:
##########
@@ -0,0 +1,1035 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp.h"
+
+#include <cmath>
+#include <cstring>
+#include <functional>
+#include <iostream>
+#include <map>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/bit_stream_utils_internal.h"
+#include "arrow/util/bpacking_internal.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo implementation (non-templated, 4 bytes)
+
+void AlpEncodedVectorInfo::Store(arrow::util::span<char> output_buffer) const {
+ ARROW_CHECK(output_buffer.size() >= GetStoredSize())
+ << "alp_vector_info_output_too_small: " << output_buffer.size() << " vs "
+ << GetStoredSize();
+
+ char* ptr = output_buffer.data();
+
+ // exponent, factor: 1 byte each
+ *ptr++ = static_cast<char>(exponent);
+ *ptr++ = static_cast<char>(factor);
+
+ // num_exceptions: 2 bytes
+ std::memcpy(ptr, &num_exceptions, sizeof(num_exceptions));
+}
+
+AlpEncodedVectorInfo AlpEncodedVectorInfo::Load(
+ arrow::util::span<const char> input_buffer) {
+ ARROW_CHECK(input_buffer.size() >= GetStoredSize())
+ << "alp_vector_info_input_too_small: " << input_buffer.size() << " vs "
+ << GetStoredSize();
+
+ AlpEncodedVectorInfo result{};
+ const char* ptr = input_buffer.data();
+
+ // exponent, factor: 1 byte each
+ result.exponent = static_cast<uint8_t>(*ptr++);
+ result.factor = static_cast<uint8_t>(*ptr++);
+
+ // num_exceptions: 2 bytes
+ std::memcpy(&result.num_exceptions, ptr, sizeof(result.num_exceptions));
+
+ return result;
+}
+
+// ----------------------------------------------------------------------
+// AlpEncodedForVectorInfo implementation (templated, 5/9 bytes)
+
+template <typename T>
+void AlpEncodedForVectorInfo<T>::Store(arrow::util::span<char> output_buffer) const {
+ ARROW_CHECK(output_buffer.size() >= GetStoredSize())
+ << "alp_for_vector_info_output_too_small: " << output_buffer.size() << " vs "
+ << GetStoredSize();
+
+ char* ptr = output_buffer.data();
+
+ // frame_of_reference: 4 bytes for float, 8 bytes for double
+ std::memcpy(ptr, &frame_of_reference, sizeof(frame_of_reference));
+ ptr += sizeof(frame_of_reference);
+
+ // bit_width: 1 byte
+ *ptr = static_cast<char>(bit_width);
+}
+
+template <typename T>
+AlpEncodedForVectorInfo<T> AlpEncodedForVectorInfo<T>::Load(
+ arrow::util::span<const char> input_buffer) {
+ ARROW_CHECK(input_buffer.size() >= GetStoredSize())
+ << "alp_for_vector_info_input_too_small: " << input_buffer.size() << " vs "
+ << GetStoredSize();
+
+ AlpEncodedForVectorInfo<T> result{};
+ const char* ptr = input_buffer.data();
+
+ // frame_of_reference: 4 bytes for float, 8 bytes for double
+ std::memcpy(&result.frame_of_reference, ptr, sizeof(result.frame_of_reference));
+ ptr += sizeof(result.frame_of_reference);
+
+ // bit_width: 1 byte
+ result.bit_width = static_cast<uint8_t>(*ptr);
+
+ return result;
+}
+
+// Explicit template instantiations for AlpEncodedForVectorInfo
+template struct AlpEncodedForVectorInfo<float>;
+template struct AlpEncodedForVectorInfo<double>;
+
+// ----------------------------------------------------------------------
+// AlpEncodedVector implementation
+
+template <typename T>
+void AlpEncodedVector<T>::Store(arrow::util::span<char> output_buffer) const {
+ const uint64_t overall_size = GetStoredSize();
+ ARROW_CHECK(output_buffer.size() >= overall_size)
+ << "alp_bit_packed_vector_store_output_too_small: " << output_buffer.size()
+ << " vs " << overall_size;
+
+ uint64_t offset = 0;
+
+ // Store AlpInfo (4 bytes)
+ alp_info.Store({output_buffer.data() + offset, AlpEncodedVectorInfo::kStoredSize});
+ offset += AlpEncodedVectorInfo::kStoredSize;
+
+ // Store ForInfo (6/10 bytes)
+ for_info.Store({output_buffer.data() + offset, AlpEncodedForVectorInfo<T>::kStoredSize});
+ offset += AlpEncodedForVectorInfo<T>::kStoredSize;
+
+ // Store data section
+ StoreDataOnly({output_buffer.data() + offset, output_buffer.size() - offset});
+}
+
+template <typename T>
+void AlpEncodedVector<T>::StoreDataOnly(arrow::util::span<char> output_buffer) const {
+ const uint64_t data_size = GetDataStoredSize();
+ ARROW_CHECK(output_buffer.size() >= data_size)
+ << "alp_bit_packed_vector_store_data_output_too_small: " << output_buffer.size()
+ << " vs " << data_size;
+
+ ARROW_CHECK(alp_info.num_exceptions == exceptions.size() &&
+ alp_info.num_exceptions == exception_positions.size())
+ << "alp_bit_packed_vector_store_num_exceptions_mismatch: "
+ << alp_info.num_exceptions << " vs " << exceptions.size() << " vs "
+ << exception_positions.size();
+
+ uint64_t offset = 0;
+
+ // Compute bit_packed_size from num_elements and bit_width
+ const uint64_t bit_packed_size =
+ AlpEncodedForVectorInfo<T>::GetBitPackedSize(num_elements, for_info.bit_width);
+
+ // Store all successfully compressed values first.
+ std::memcpy(output_buffer.data() + offset, packed_values.data(), bit_packed_size);
+ offset += bit_packed_size;
+
+ // Store exception positions.
+ const uint64_t exception_position_size =
+ alp_info.num_exceptions * sizeof(AlpConstants::PositionType);
+ std::memcpy(output_buffer.data() + offset, exception_positions.data(),
+ exception_position_size);
+ offset += exception_position_size;
+
+ // Store exception values.
+ const uint64_t exception_size = alp_info.num_exceptions * sizeof(T);
+ std::memcpy(output_buffer.data() + offset, exceptions.data(), exception_size);
+ offset += exception_size;
+
+ ARROW_CHECK(offset == data_size)
+ << "alp_bit_packed_vector_data_size_mismatch: " << offset << " vs " << data_size;
+}
+
+template <typename T>
+AlpEncodedVector<T> AlpEncodedVector<T>::Load(
+ arrow::util::span<const char> input_buffer, uint16_t num_elements) {
+ ARROW_CHECK(num_elements <= AlpConstants::kAlpVectorSize)
+ << "alp_compression_state_element_count_too_large: " << num_elements << " vs "
+ << AlpConstants::kAlpVectorSize;
+
+ AlpEncodedVector<T> result;
+ uint64_t input_offset = 0;
+
+ // Load AlpInfo (4 bytes)
+ result.alp_info = AlpEncodedVectorInfo::Load(
+ {input_buffer.data() + input_offset, AlpEncodedVectorInfo::kStoredSize});
+ input_offset += AlpEncodedVectorInfo::kStoredSize;
+
+ // Load ForInfo (6/10 bytes)
+ result.for_info = AlpEncodedForVectorInfo<T>::Load(
+ {input_buffer.data() + input_offset, AlpEncodedForVectorInfo<T>::kStoredSize});
+ input_offset += AlpEncodedForVectorInfo<T>::kStoredSize;
+
+ result.num_elements = num_elements;
+
+ const uint64_t overall_size =
+ GetStoredSize(result.alp_info, result.for_info, num_elements);
+
+ ARROW_CHECK(input_buffer.size() >= overall_size)
+ << "alp_compression_state_input_too_small: " << input_buffer.size() << " vs "
+ << overall_size;
+
+ // Compute bit_packed_size from num_elements and bit_width
+ const uint64_t bit_packed_size =
+ AlpEncodedForVectorInfo<T>::GetBitPackedSize(num_elements, result.for_info.bit_width);
+
+ // Optimization: Use UnsafeResize to avoid zero-initialization before memcpy.
+ // This is safe for POD types since we immediately overwrite with memcpy.
+ result.packed_values.UnsafeResize(bit_packed_size);
+ std::memcpy(result.packed_values.data(), input_buffer.data() + input_offset,
+ bit_packed_size);
+ input_offset += bit_packed_size;
+
+ result.exception_positions.UnsafeResize(result.alp_info.num_exceptions);
+ const uint64_t exception_position_size =
+ result.alp_info.num_exceptions * sizeof(AlpConstants::PositionType);
+ std::memcpy(result.exception_positions.data(), input_buffer.data() + input_offset,
+ exception_position_size);
+ input_offset += exception_position_size;
+
+ result.exceptions.UnsafeResize(result.alp_info.num_exceptions);
+ const uint64_t exception_size = result.alp_info.num_exceptions * sizeof(T);
+ std::memcpy(result.exceptions.data(), input_buffer.data() + input_offset,
+ exception_size);
+ return result;
+}
+
+template <typename T>
+uint64_t AlpEncodedVector<T>::GetStoredSize() const {
+ return GetStoredSize(alp_info, for_info, num_elements);
+}
+
+template <typename T>
+uint64_t AlpEncodedVector<T>::GetStoredSize(const AlpEncodedVectorInfo& alp_info,
+ const AlpEncodedForVectorInfo<T>& for_info,
+ uint16_t num_elements) {
+ const uint64_t bit_packed_size =
+ AlpEncodedForVectorInfo<T>::GetBitPackedSize(num_elements, for_info.bit_width);
+ return AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize +
+ bit_packed_size +
+ alp_info.num_exceptions * (sizeof(AlpConstants::PositionType) + sizeof(T));
+}
+
+template <typename T>
+bool AlpEncodedVector<T>::operator==(const AlpEncodedVector<T>& other) const {
+ if (alp_info != other.alp_info || for_info != other.for_info ||
+ num_elements != other.num_elements) {
+ return false;
+ }
+ if (packed_values.size() != other.packed_values.size() ||
+ !std::equal(packed_values.begin(), packed_values.end(), other.packed_values.begin())) {
+ return false;
+ }
+ if (exceptions.size() != other.exceptions.size() ||
+ !std::equal(exceptions.begin(), exceptions.end(), other.exceptions.begin())) {
+ return false;
+ }
+ if (exception_positions.size() != other.exception_positions.size() ||
+ !std::equal(exception_positions.begin(), exception_positions.end(),
+ other.exception_positions.begin())) {
+ return false;
+ }
+ return true;
+}
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorView implementation
+
+template <typename T>
+AlpEncodedVectorView<T> AlpEncodedVectorView<T>::LoadView(
+ arrow::util::span<const char> input_buffer, uint16_t num_elements) {
+ ARROW_CHECK(num_elements <= AlpConstants::kAlpVectorSize)
+ << "alp_view_element_count_too_large: " << num_elements << " vs "
+ << AlpConstants::kAlpVectorSize;
+
+ AlpEncodedVectorView<T> result;
+ uint64_t input_offset = 0;
+
+ // Load AlpInfo (4 bytes)
+ result.alp_info = AlpEncodedVectorInfo::Load(
+ {input_buffer.data() + input_offset, AlpEncodedVectorInfo::kStoredSize});
+ input_offset += AlpEncodedVectorInfo::kStoredSize;
+
+ // Load ForInfo (6/10 bytes)
+ result.for_info = AlpEncodedForVectorInfo<T>::Load(
+ {input_buffer.data() + input_offset, AlpEncodedForVectorInfo<T>::kStoredSize});
+ input_offset += AlpEncodedForVectorInfo<T>::kStoredSize;
+
+ result.num_elements = num_elements;
+
+ const uint64_t overall_size =
+ AlpEncodedVector<T>::GetStoredSize(result.alp_info, result.for_info, num_elements);
+
+ ARROW_CHECK(input_buffer.size() >= overall_size)
+ << "alp_view_input_too_small: " << input_buffer.size() << " vs " << overall_size;
+
+ // Load data section (after metadata)
+ AlpEncodedVectorView<T> data_view = LoadViewDataOnly(
+ {input_buffer.data() + input_offset, input_buffer.size() - input_offset},
+ result.alp_info, result.for_info, num_elements);
+
+ // Copy the loaded data into result
+ result.packed_values = data_view.packed_values;
+ result.exception_positions = std::move(data_view.exception_positions);
+ result.exceptions = std::move(data_view.exceptions);
+
+ return result;
+}
+
+template <typename T>
+AlpEncodedVectorView<T> AlpEncodedVectorView<T>::LoadViewDataOnly(
+ arrow::util::span<const char> input_buffer, const AlpEncodedVectorInfo& alp_info,
+ const AlpEncodedForVectorInfo<T>& for_info, uint16_t num_elements) {
+ ARROW_CHECK(num_elements <= AlpConstants::kAlpVectorSize)
+ << "alp_view_data_only_element_count_too_large: " << num_elements << " vs "
+ << AlpConstants::kAlpVectorSize;
+
+ AlpEncodedVectorView<T> result;
+ result.alp_info = alp_info;
+ result.for_info = for_info;
+ result.num_elements = num_elements;
+
+ const uint64_t data_size = for_info.GetDataStoredSize(num_elements, alp_info.num_exceptions);
+ ARROW_CHECK(input_buffer.size() >= data_size)
+ << "alp_view_data_only_input_too_small: " << input_buffer.size() << " vs "
+ << data_size;
+
+ uint64_t input_offset = 0;
+
+ // Compute bit_packed_size from num_elements and bit_width
+ const uint64_t bit_packed_size =
+ AlpEncodedForVectorInfo<T>::GetBitPackedSize(num_elements, for_info.bit_width);
+
+ // Zero-copy for packed values (bytes have no alignment requirements)
+ result.packed_values = {
+ reinterpret_cast<const uint8_t*>(input_buffer.data() + input_offset),
+ bit_packed_size};
+ input_offset += bit_packed_size;
+
+ // Copy exception positions into aligned storage to avoid UB from misaligned access.
+ // Exceptions are rare (typically < 5%), so the copy overhead is negligible.
+ const uint64_t exception_position_size =
+ alp_info.num_exceptions * sizeof(AlpConstants::PositionType);
+ result.exception_positions.UnsafeResize(alp_info.num_exceptions);
+ std::memcpy(result.exception_positions.data(), input_buffer.data() + input_offset,
+ exception_position_size);
+ input_offset += exception_position_size;
+
+ // Copy exception values into aligned storage to avoid UB from misaligned access.
+ const uint64_t exception_size = alp_info.num_exceptions * sizeof(T);
+ result.exceptions.UnsafeResize(alp_info.num_exceptions);
+ std::memcpy(result.exceptions.data(), input_buffer.data() + input_offset,
+ exception_size);
+
+ return result;
+}
+
+template <typename T>
+uint64_t AlpEncodedVectorView<T>::GetStoredSize() const {
+ return AlpEncodedVector<T>::GetStoredSize(alp_info, for_info, num_elements);
+}
+
+template struct AlpEncodedVectorView<float>;
+template struct AlpEncodedVectorView<double>;
+
+// ----------------------------------------------------------------------
+// AlpMetadataCache implementation
+
+template <typename T>
+AlpMetadataCache<T> AlpMetadataCache<T>::Load(
+ uint32_t num_vectors, uint32_t vector_size, uint32_t total_elements,
+ AlpIntegerEncoding integer_encoding,
+ arrow::util::span<const char> alp_metadata_buffer,
+ arrow::util::span<const char> int_encoding_metadata_buffer) {
+ AlpMetadataCache<T> cache;
+
+ if (num_vectors == 0) {
+ return cache;
+ }
+
+ const uint64_t alp_info_size = AlpEncodedVectorInfo::kStoredSize;
+ const uint64_t expected_alp_size = num_vectors * alp_info_size;
+
+ ARROW_CHECK(alp_metadata_buffer.size() >= expected_alp_size)
+ << "alp_metadata_cache_alp_buffer_too_small: " << alp_metadata_buffer.size()
+ << " vs " << expected_alp_size;
+
+ cache.alp_infos_.reserve(num_vectors);
+ cache.cumulative_data_offsets_.reserve(num_vectors);
+ cache.vector_num_elements_.reserve(num_vectors);
+
+ // Calculate number of full vectors and remainder
+ const uint32_t num_full_vectors = total_elements / vector_size;
+ const uint32_t remainder = total_elements % vector_size;
+
+ // Load integer encoding metadata based on encoding type
+ switch (integer_encoding) {
+ case AlpIntegerEncoding::kForBitPack: {
+ const uint64_t for_info_size = AlpEncodedForVectorInfo<T>::kStoredSize;
+ const uint64_t expected_for_size = num_vectors * for_info_size;
+ ARROW_CHECK(int_encoding_metadata_buffer.size() >= expected_for_size)
+ << "alp_metadata_cache_for_buffer_too_small: "
+ << int_encoding_metadata_buffer.size() << " vs " << expected_for_size;
+ cache.for_infos_.reserve(num_vectors);
+
+ uint64_t cumulative_offset = 0;
+ for (uint32_t i = 0; i < num_vectors; i++) {
+ // Load AlpInfo
+ const AlpEncodedVectorInfo alp_info = AlpEncodedVectorInfo::Load(
+ {alp_metadata_buffer.data() + i * alp_info_size, alp_info_size});
+ cache.alp_infos_.push_back(alp_info);
+
+ // Load ForInfo for kForBitPack encoding
+ const AlpEncodedForVectorInfo<T> for_info = AlpEncodedForVectorInfo<T>::Load(
+ {int_encoding_metadata_buffer.data() + i * for_info_size, for_info_size});
+ cache.for_infos_.push_back(for_info);
+
+ // Calculate number of elements for this vector
+ const uint16_t this_vector_elements =
+ (i < num_full_vectors) ? static_cast<uint16_t>(vector_size)
+ : static_cast<uint16_t>(remainder);
+ cache.vector_num_elements_.push_back(this_vector_elements);
+
+ // Store cumulative offset (offset to start of this vector's data)
+ cache.cumulative_data_offsets_.push_back(cumulative_offset);
+
+ // Advance offset by this vector's data size
+ cumulative_offset +=
+ for_info.GetDataStoredSize(this_vector_elements, alp_info.num_exceptions);
+ }
+ cache.total_data_size_ = cumulative_offset;
+ } break;
+
+ default:
+ ARROW_CHECK(false) << "unsupported_integer_encoding: "
+ << static_cast<int>(integer_encoding);
+ break;
+ }
+
+ return cache;
+}
+
+template class AlpMetadataCache<float>;
+template class AlpMetadataCache<double>;
+
+template class AlpEncodedVector<float>;
+template class AlpEncodedVector<double>;
+
+// ----------------------------------------------------------------------
+// Internal helper classes
+
+namespace {
+
+/// \brief Helper class for encoding/decoding individual values
+template <typename T>
+class AlpInlines : private AlpConstants {
+ public:
+ using Constants = AlpTypedConstants<T>;
+ using ExactType = typename Constants::FloatingToExact;
+ using SignedExactType = typename Constants::FloatingToSignedExact;
+
+ /// \brief Check if float is a special value that cannot be converted
+ static inline bool IsImpossibleToEncode(const T n) {
+ // We do not have to check for positive or negative infinity, since
+ // std::numeric_limits<T>::infinity() > std::numeric_limits<T>::max()
+ // and vice versa for negative infinity.
+ return std::isnan(n) || n > Constants::kEncodingUpperLimit ||
+ n < Constants::kEncodingLowerLimit ||
+ (n == 0.0 && std::signbit(n)); // Verification for -0.0
+ }
+
+ /// \brief Convert a float to an int without rounding
+ static inline auto FastRound(T n) -> SignedExactType {
+ n = n + Constants::kMagicNumber - Constants::kMagicNumber;
+ return static_cast<SignedExactType>(n);
+ }
+
+ /// \brief Fast way to round float to nearest integer
+ static inline auto NumberToInt(T n) -> SignedExactType {
+ if (IsImpossibleToEncode(n)) {
+ return static_cast<SignedExactType>(Constants::kEncodingUpperLimit);
+ }
+ return FastRound(n);
+ }
+
+ /// \brief Convert a float into an int using encoding options
+ static inline SignedExactType EncodeValue(
+ const T value, const AlpExponentAndFactor exponent_and_factor) {
+ const T tmp_encoded_value = value *
+ Constants::GetExponent(exponent_and_factor.exponent) *
+ Constants::GetFactor(exponent_and_factor.factor);
+ return NumberToInt(tmp_encoded_value);
+ }
+
+ /// \brief Reconvert an int to a float using encoding options
+ static inline T DecodeValue(const SignedExactType encoded_value,
+ const AlpExponentAndFactor exponent_and_factor) {
+ // The cast to T is needed to prevent a signed integer overflow.
+ return static_cast<T>(encoded_value) * GetFactor(exponent_and_factor.factor) *
+ Constants::GetFactor(exponent_and_factor.exponent);
+ }
+};
+
+/// \brief Helper struct for tracking compression combinations
+struct AlpCombination {
+ AlpExponentAndFactor exponent_and_factor;
+ uint64_t num_appearances{0};
+ uint64_t estimated_compression_size{0};
+};
+
+/// \brief Compare two ALP combinations to determine which is better
+///
+/// Return true if c1 is a better combination than c2.
+/// First criteria is number of times it appears as best combination.
+/// Second criteria is the estimated compression size.
+/// Third criteria is bigger exponent.
+/// Fourth criteria is bigger factor.
+bool CompareAlpCombinations(const AlpCombination& c1, const AlpCombination& c2) {
+ return (c1.num_appearances > c2.num_appearances) ||
+ (c1.num_appearances == c2.num_appearances &&
+ (c1.estimated_compression_size < c2.estimated_compression_size)) ||
+ ((c1.num_appearances == c2.num_appearances &&
+ c1.estimated_compression_size == c2.estimated_compression_size) &&
+ (c2.exponent_and_factor.exponent < c1.exponent_and_factor.exponent)) ||
+ ((c1.num_appearances == c2.num_appearances &&
+ c1.estimated_compression_size == c2.estimated_compression_size &&
+ c2.exponent_and_factor.exponent == c1.exponent_and_factor.exponent) &&
+ (c2.exponent_and_factor.factor < c1.exponent_and_factor.factor));
+}
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCompression implementation
+
+template <typename T>
+std::optional<uint64_t> AlpCompression<T>::EstimateCompressedSize(
+ const std::vector<T>& input_vector,
+ const AlpExponentAndFactor exponent_and_factor,
+ const bool penalize_exceptions) {
+ // Dry compress a vector (ideally a sample) to estimate ALP compression size
+ // given an exponent and factor.
+ SignedExactType max_encoded_value = std::numeric_limits<SignedExactType>::min();
+ SignedExactType min_encoded_value = std::numeric_limits<SignedExactType>::max();
+
+ uint64_t num_exceptions = 0;
+ uint64_t num_non_exceptions = 0;
+ for (const T& value : input_vector) {
+ const SignedExactType encoded_value =
+ AlpInlines<T>::EncodeValue(value, exponent_and_factor);
+ T decoded_value = AlpInlines<T>::DecodeValue(encoded_value, exponent_and_factor);
+ if (decoded_value == value) {
+ num_non_exceptions++;
+ max_encoded_value = std::max(encoded_value, max_encoded_value);
+ min_encoded_value = std::min(encoded_value, min_encoded_value);
+ continue;
+ }
+ num_exceptions++;
+ }
+
+ // We penalize combinations which yield almost all exceptions.
+ if (penalize_exceptions && num_non_exceptions < 2) {
+ return std::nullopt;
+ }
+
+ // Evaluate factor/exponent compression size (we optimize for FOR).
+ const ExactType delta = (static_cast<ExactType>(max_encoded_value) -
+ static_cast<ExactType>(min_encoded_value));
+
+ const uint32_t estimated_bits_per_value =
+ static_cast<uint32_t>(std::ceil(std::log2(delta + 1)));
+ uint64_t estimated_compression_size = input_vector.size() * estimated_bits_per_value;
+ estimated_compression_size +=
+ num_exceptions * (kExactTypeBitSize + (sizeof(PositionType) * 8));
+ return estimated_compression_size;
+}
+
+template <typename T>
+AlpEncodingPreset AlpCompression<T>::CreateEncodingPreset(
+ const std::vector<std::vector<T>>& vectors_sampled) {
+ // Find the best combinations of factor-exponent from each sampled vector.
+ // This function is called once per segment.
+ // This operates over ALP first level samples.
+ static constexpr uint64_t kMaxCombinationCount =
+ (Constants::kMaxExponent + 1) * (Constants::kMaxExponent + 2) / 2;
+
+ std::map<AlpExponentAndFactor, uint64_t> best_k_combinations_hash;
+
+ uint64_t best_compressed_size_bits = std::numeric_limits<uint64_t>::max();
+ // For each vector sampled.
+ for (const std::vector<T>& sampled_vector : vectors_sampled) {
+ const uint64_t num_samples = sampled_vector.size();
+ const AlpExponentAndFactor best_encoding_options{Constants::kMaxExponent,
+ Constants::kMaxExponent};
+
+ // Start optimization with worst possible total bits from compression.
+ const uint64_t best_total_bits =
+ (num_samples * (kExactTypeBitSize + sizeof(PositionType) * 8)) +
+ (num_samples * kExactTypeBitSize);
+
+ // N of appearances is irrelevant at this phase; we search for best compression.
+ AlpCombination best_combination{best_encoding_options, 0, best_total_bits};
+ // Try all combinations to find the one which minimizes compression size.
+ for (uint8_t exp_idx = 0; exp_idx <= Constants::kMaxExponent; exp_idx++) {
+ for (uint8_t factor_idx = 0; factor_idx <= exp_idx; factor_idx++) {
+ const AlpExponentAndFactor current_exponent_and_factor{exp_idx, factor_idx};
+ std::optional<uint64_t> estimated_compression_size = EstimateCompressedSize(
+ sampled_vector, current_exponent_and_factor, /*penalize_exceptions=*/true);
+
+ // Skip comparison for values that are not compressible.
+ if (!estimated_compression_size.has_value()) {
+ continue;
+ }
+
+ const AlpCombination current_combination{current_exponent_and_factor, 0,
+ *estimated_compression_size};
+ if (CompareAlpCombinations(current_combination, best_combination)) {
+ best_combination = current_combination;
+ best_compressed_size_bits =
+ std::min(best_compressed_size_bits, *estimated_compression_size);
+ }
+ }
+ }
+ best_k_combinations_hash[best_combination.exponent_and_factor]++;
+ }
+
+ // Convert our hash to a Combination vector to be able to sort.
+ // Note that this vector should mostly be small (< 10 combinations).
+ std::vector<AlpCombination> best_k_combinations;
+ best_k_combinations.reserve(
+ std::min(best_k_combinations_hash.size(), kMaxCombinationCount));
+ for (const auto& combination : best_k_combinations_hash) {
+ best_k_combinations.emplace_back(AlpCombination{
+ combination.first, // Encoding Indices
+ combination.second, // N of times it appeared (hash value)
+ 0 // Compression size is irrelevant since we compare different vectors.
+ });
+ }
+ std::sort(best_k_combinations.begin(), best_k_combinations.end(),
+ CompareAlpCombinations);
+
+ std::vector<AlpExponentAndFactor> combinations;
+ // Save k' best combinations.
+ for (uint64_t i = 0;
+ i < std::min(kMaxCombinations, static_cast<uint8_t>(best_k_combinations.size()));
+ i++) {
+ combinations.push_back(best_k_combinations[i].exponent_and_factor);
+ }
+
+ const uint64_t best_compressed_size_bytes =
+ std::ceil(best_compressed_size_bits / 8.0);
+ return {combinations, best_compressed_size_bytes};
+}
+
+template <typename T>
+std::vector<T> AlpCompression<T>::CreateSample(arrow::util::span<const T> input) {
+ // Sample equidistant values within a vector; skip a fixed number of values.
+ const auto idx_increments = std::max<uint32_t>(
+ 1, static_cast<uint32_t>(std::ceil(static_cast<double>(input.size()) /
+ AlpConstants::kSamplerSamplesPerVector)));
+ std::vector<T> vector_sample;
+ vector_sample.reserve(std::ceil(input.size() / static_cast<double>(idx_increments)));
+ for (uint64_t i = 0; i < input.size(); i += idx_increments) {
+ vector_sample.push_back(input[i]);
+ }
+ return vector_sample;
+}
+
+template <typename T>
+AlpExponentAndFactor AlpCompression<T>::FindBestExponentAndFactor(
+ arrow::util::span<const T> input,
+ const std::vector<AlpExponentAndFactor>& combinations) {
+ // Find the best factor-exponent combination from within the best k combinations.
+ // This is ALP second level sampling.
+ if (combinations.size() == 1) {
+ return combinations.front();
+ }
+
+ const std::vector<T> sample_vector = CreateSample(input);
+
+ AlpExponentAndFactor best_exponent_and_factor;
+ uint64_t best_total_bits = std::numeric_limits<uint64_t>::max();
+ uint64_t worse_total_bits_counter = 0;
+
+ // Try each K combination to find the one which minimizes compression size.
+ for (const AlpExponentAndFactor& exponent_and_factor : combinations) {
+ std::optional<uint64_t> estimated_compression_size = EstimateCompressedSize(
+ sample_vector, exponent_and_factor, /*penalize_exceptions=*/false);
+
+ // Skip exponents and factors which result in many exceptions.
+ if (!estimated_compression_size.has_value()) {
+ continue;
+ }
+
+ // If current compression size is worse or equal than current best combination.
+ if (estimated_compression_size >= best_total_bits) {
+ worse_total_bits_counter += 1;
+ // Early exit strategy.
+ if (worse_total_bits_counter == kSamplingEarlyExitThreshold) {
+ break;
+ }
+ continue;
+ }
+ // Otherwise replace the best and continue trying with next combination.
+ best_total_bits = estimated_compression_size.value();
+ best_exponent_and_factor = exponent_and_factor;
+ worse_total_bits_counter = 0;
+ }
+ return best_exponent_and_factor;
+}
+
+template <typename T>
+auto AlpCompression<T>::EncodeVector(arrow::util::span<const T> input_vector,
+ AlpExponentAndFactor exponent_and_factor)
+ -> EncodingResult {
+ arrow::internal::StaticVector<SignedExactType, kAlpVectorSize> encoded_integers;
+ arrow::internal::StaticVector<T, kAlpVectorSize> exceptions;
+ arrow::internal::StaticVector<PositionType, kAlpVectorSize> exception_positions;
+
+ // Encoding Float/Double to SignedExactType(Int32, Int64).
+ // Encode all values regardless of correctness to recover original floating-point.
+ uint64_t input_offset = 0;
+ for (const T input : input_vector) {
+ const SignedExactType encoded_value =
+ AlpInlines<T>::EncodeValue(input, exponent_and_factor);
+ const T decoded_value = AlpInlines<T>::DecodeValue(encoded_value, exponent_and_factor);
+ encoded_integers.push_back(encoded_value);
+ // Detect exceptions using a predicated comparison.
+ if (decoded_value != input) {
+ exception_positions.push_back(input_offset);
+ }
+ input_offset++;
+ }
+
+ // Finding first non-exception value.
+ SignedExactType first_non_exception_value = 0;
+ PositionType exception_offset = 0;
+ for (const PositionType exception_position : exception_positions) {
+ if (exception_offset != exception_position) {
+ first_non_exception_value = encoded_integers[exception_offset];
+ break;
+ }
+ exception_offset++;
+ }
+
+ // Use first non-exception value as placeholder for all exception values.
+ for (const PositionType exception_position : exception_positions) {
+ const T actual_value = input_vector[exception_position];
+ encoded_integers[exception_position] = first_non_exception_value;
+ exceptions.push_back(actual_value);
+ }
+
+ // Analyze FOR.
+ const auto [min, max] =
+ std::minmax_element(encoded_integers.begin(), encoded_integers.end());
+ const auto frame_of_reference = static_cast<ExactType>(*min);
+
+ for (SignedExactType& encoded_integer : encoded_integers) {
+ // Use SafeCopy to avoid strict aliasing violation when converting between
+ // signed and unsigned integer types of the same size.
+ ExactType unsigned_value = util::SafeCopy<ExactType>(encoded_integer);
+ unsigned_value -= frame_of_reference;
+ encoded_integer = util::SafeCopy<SignedExactType>(unsigned_value);
+ }
+
+ const ExactType min_max_diff =
+ (static_cast<ExactType>(*max) - static_cast<ExactType>(*min));
+ return EncodingResult{encoded_integers, exceptions, exception_positions, min_max_diff,
+ frame_of_reference};
+}
+
+template <typename T>
+auto AlpCompression<T>::BitPackIntegers(
+ arrow::util::span<const SignedExactType> integers, const uint64_t min_max_diff)
+ -> BitPackingResult {
+ uint8_t bit_width = 0;
+
+ if (min_max_diff == 0) {
+ bit_width = 0;
+ } else if constexpr (std::is_same_v<T, float>) {
+ bit_width = sizeof(T) * 8 - __builtin_clz(min_max_diff);
+ } else if constexpr (std::is_same_v<T, double>) {
+ bit_width = sizeof(T) * 8 - __builtin_clzll(min_max_diff);
+ }
+ const uint16_t bit_packed_size =
+ static_cast<uint16_t>(std::ceil((bit_width * integers.size()) / 8.0));
+
+ arrow::internal::StaticVector<uint8_t, kAlpVectorSize * sizeof(T)> packed_integers;
+ // Use unsafe resize to avoid zero-initialization. Zero initialization was
+ // resulting in around 2-3% degradation in compression speed.
+ packed_integers.UnsafeResize(bit_packed_size);
+ if (bit_width > 0) { // Only execute BP if writing data.
+ // Use Arrow's BitWriter for packing (loop-based).
+ arrow::bit_util::BitWriter writer(packed_integers.data(),
+ static_cast<int>(bit_packed_size));
+ for (uint64_t i = 0; i < integers.size(); ++i) {
+ writer.PutValue(static_cast<uint64_t>(integers[i]), bit_width);
+ }
+ writer.Flush(false);
+ }
+ return {packed_integers, bit_width, bit_packed_size};
+}
+
+template <typename T>
+AlpEncodedVector<T> AlpCompression<T>::CompressVector(const T* input_vector,
+ uint16_t num_elements,
+ const AlpEncodingPreset& preset) {
+ // Compress by finding a fitting exponent/factor, encode input, and bitpack.
+ const arrow::util::span<const T> input_span{input_vector, num_elements};
+ const AlpExponentAndFactor exponent_and_factor =
+ FindBestExponentAndFactor(input_span, preset.combinations);
+ const EncodingResult encoding_result = EncodeVector(input_span, exponent_and_factor);
+ BitPackingResult bitpacking_result;
+ switch (preset.integer_encoding) {
+ case AlpIntegerEncoding::kForBitPack:
+ bitpacking_result =
+ BitPackIntegers(encoding_result.encoded_integers, encoding_result.min_max_diff);
+ break;
+ default:
+ ARROW_CHECK(false) << "invalid_integer_encoding: "
+ << static_cast<int>(preset.integer_encoding);
+ break;
+ }
+
+ // Build the result with split metadata
+ AlpEncodedVector<T> result;
+
+ // ALP metadata (4 bytes)
+ result.alp_info.exponent = exponent_and_factor.exponent;
+ result.alp_info.factor = exponent_and_factor.factor;
+ result.alp_info.num_exceptions =
+ static_cast<uint16_t>(encoding_result.exceptions.size());
+
+ // FOR metadata (5/9 bytes)
+ result.for_info.frame_of_reference = encoding_result.frame_of_reference;
+ result.for_info.bit_width = bitpacking_result.bit_width;
+
+ result.num_elements = num_elements;
+ result.packed_values = bitpacking_result.packed_integers;
+ result.exceptions = encoding_result.exceptions;
+ result.exception_positions = encoding_result.exception_positions;
+ return result;
+}
+
+template <typename T>
+auto AlpCompression<T>::BitUnpackIntegers(
+ arrow::util::span<const uint8_t> packed_integers,
+ const AlpEncodedForVectorInfo<T>& for_info, uint16_t num_elements)
+ -> arrow::internal::StaticVector<ExactType, kAlpVectorSize> {
+ arrow::internal::StaticVector<ExactType, kAlpVectorSize> encoded_integers;
+ // Optimization: Use UnsafeResize to avoid zero-initialization.
+ // Safe because we immediately write to all elements via unpack().
+ encoded_integers.UnsafeResize(num_elements);
+
+ if (for_info.bit_width > 0) {
+ // Arrow's SIMD unpack works in fixed batch sizes. All SIMD implementations
+ // (SIMD128/NEON, SIMD256/AVX2, SIMD512/AVX512) have identical batch sizes:
+ // - uint32_t (float): Simd*UnpackerForWidth::kValuesUnpacked = 32
+ // - uint64_t (double): Simd*UnpackerForWidth::kValuesUnpacked = 64
+ // These constants are in anonymous namespaces (internal implementation detail),
+ // so we hardcode them here.
+ constexpr int kMinBatchSize = std::is_same_v<T, float> ? 32 : 64;
+ const int num_elems = static_cast<int>(num_elements);
+ const int num_complete_batches = num_elems / kMinBatchSize;
+ const int num_complete_elements = num_complete_batches * kMinBatchSize;
+
+ // Use Arrow's SIMD-optimized unpack for complete batches.
+ if (num_complete_elements > 0) {
+ arrow::internal::unpack(packed_integers.data(), encoded_integers.data(),
+ num_complete_elements, for_info.bit_width);
+ }
+
+ // Handle remaining elements (<64) with BitReader to match BitWriter format.
+ const int remaining = num_elems - num_complete_elements;
+ if (remaining > 0) {
+ // Calculate byte offset where SIMD unpack finished
+ const uint64_t bits_consumed_by_simd =
+ static_cast<uint64_t>(num_complete_elements) * for_info.bit_width;
+ // Round up to next byte
+ const uint64_t bytes_consumed_by_simd = (bits_consumed_by_simd + 7) / 8;
+
+ // Use BitReader for remaining elements starting from where SIMD left off
+ arrow::bit_util::BitReader reader(
+ packed_integers.data() + bytes_consumed_by_simd,
+ static_cast<int>(packed_integers.size() - bytes_consumed_by_simd));
+
+ for (int i = 0; i < remaining; ++i) {
+ uint64_t value = 0;
+ if (reader.GetValue(for_info.bit_width, &value)) {
+ encoded_integers[num_complete_elements + i] = static_cast<ExactType>(value);
+ } else {
+ encoded_integers[num_complete_elements + i] = 0;
+ }
+ }
+ }
+ } else {
+ std::memset(encoded_integers.data(), 0, num_elements * sizeof(ExactType));
+ }
+ return encoded_integers;
+}
+
+template <typename T>
+template <typename TargetType>
+void AlpCompression<T>::DecodeVector(TargetType* output_vector,
+ arrow::util::span<ExactType> input_vector,
+ const AlpEncodedVectorInfo& alp_info,
+ const AlpEncodedForVectorInfo<T>& for_info,
+ uint16_t num_elements) {
+ // Fused unFOR + decode loop - reduces memory traffic by avoiding
+ // intermediate write-then-read of the unFOR'd values.
+ const ExactType* data = input_vector.data();
+ const ExactType frame_of_ref = for_info.frame_of_reference;
+
+#pragma GCC unroll AlpConstants::kLoopUnrolls
Review Comment:
Yeah, I think my main issue here is code consistency/maintainability. I think xsimd might be the preferred maintenance route. I will yield to @pitrou for his guidance on how we should structure these optimizations.
##########
cpp/src/arrow/util/alp/alp.cc:
##########
@@ -0,0 +1,1039 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp.h"
+
+#include <cmath>
+#include <cstring>
+#include <functional>
+#include <iostream>
+#include <map>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/bit_stream_utils_internal.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/bpacking_internal.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (frame_of_reference,
+// num_exceptions, offsets) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo implementation (non-templated, 4 bytes)
+
+void AlpEncodedVectorInfo::Store(arrow::util::span<char> output_buffer) const {
+ ARROW_CHECK(output_buffer.size() >= GetStoredSize())
+ << "alp_vector_info_output_too_small: " << output_buffer.size() << " vs "
+ << GetStoredSize();
+
+ char* ptr = output_buffer.data();
+
+ // exponent, factor: 1 byte each
+ *ptr++ = static_cast<char>(exponent);
+ *ptr++ = static_cast<char>(factor);
+
+ // num_exceptions: 2 bytes
+ std::memcpy(ptr, &num_exceptions, sizeof(num_exceptions));
+}
+
+AlpEncodedVectorInfo AlpEncodedVectorInfo::Load(
+ arrow::util::span<const char> input_buffer) {
+ ARROW_CHECK(input_buffer.size() >= GetStoredSize())
Review Comment:
I feel like this check should be made clear in the header,
##########
cpp/src/arrow/util/alp/alp_wrapper.h:
##########
Review Comment:
rename to alp_codec.h (and .cc above)>
##########
cpp/src/arrow/util/alp/alp.cc:
##########
@@ -0,0 +1,1039 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp.h"
+
+#include <cmath>
+#include <cstring>
+#include <functional>
+#include <iostream>
+#include <map>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/bit_stream_utils_internal.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/bpacking_internal.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (frame_of_reference,
+// num_exceptions, offsets) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo implementation (non-templated, 4 bytes)
+
+void AlpEncodedVectorInfo::Store(arrow::util::span<char> output_buffer) const {
+ ARROW_CHECK(output_buffer.size() >= GetStoredSize())
+ << "alp_vector_info_output_too_small: " << output_buffer.size() << " vs "
+ << GetStoredSize();
+
+ char* ptr = output_buffer.data();
+
+ // exponent, factor: 1 byte each
+ *ptr++ = static_cast<char>(exponent);
+ *ptr++ = static_cast<char>(factor);
+
+ // num_exceptions: 2 bytes
+ std::memcpy(ptr, &num_exceptions, sizeof(num_exceptions));
Review Comment:
safe load?
##########
cpp/src/arrow/util/alp/alp.cc:
##########
@@ -0,0 +1,1039 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp.h"
+
+#include <cmath>
+#include <cstring>
+#include <functional>
+#include <iostream>
+#include <map>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/bit_stream_utils_internal.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/bpacking_internal.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (frame_of_reference,
+// num_exceptions, offsets) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo implementation (non-templated, 4 bytes)
+
+void AlpEncodedVectorInfo::Store(arrow::util::span<char> output_buffer) const {
+ ARROW_CHECK(output_buffer.size() >= GetStoredSize())
Review Comment:
If this is a precondition we should clearly state it in the contract in the header.
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
+ header.num_elements = static_cast<uint32_t>(element_count);
+
+ std::memcpy(encoded_header, &header.compression_mode, 3);
+ std::memcpy(encoded_header + 3, &header.num_elements, sizeof(header.num_elements));
Review Comment:
safe load?
##########
cpp/src/arrow/util/alp/alp.cc:
##########
@@ -0,0 +1,1039 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp.h"
+
+#include <cmath>
+#include <cstring>
+#include <functional>
+#include <iostream>
+#include <map>
+
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/bit_stream_utils_internal.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/bpacking_internal.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/small_vector.h"
+#include "arrow/util/span.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (frame_of_reference,
+// num_exceptions, offsets) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+// ----------------------------------------------------------------------
+// AlpEncodedVectorInfo implementation (non-templated, 4 bytes)
+
+void AlpEncodedVectorInfo::Store(arrow::util::span<char> output_buffer) const {
+ ARROW_CHECK(output_buffer.size() >= GetStoredSize())
+ << "alp_vector_info_output_too_small: " << output_buffer.size() << " vs "
+ << GetStoredSize();
+
+ char* ptr = output_buffer.data();
+
+ // exponent, factor: 1 byte each
+ *ptr++ = static_cast<char>(exponent);
+ *ptr++ = static_cast<char>(factor);
+
+ // num_exceptions: 2 bytes
+ std::memcpy(ptr, &num_exceptions, sizeof(num_exceptions));
+}
+
+AlpEncodedVectorInfo AlpEncodedVectorInfo::Load(
+ arrow::util::span<const char> input_buffer) {
+ ARROW_CHECK(input_buffer.size() >= GetStoredSize())
+ << "alp_vector_info_input_too_small: " << input_buffer.size() << " vs "
+ << GetStoredSize();
+
+ AlpEncodedVectorInfo result{};
+ const char* ptr = input_buffer.data();
+
+ // exponent, factor: 1 byte each
+ result.exponent = static_cast<uint8_t>(*ptr++);
+ result.factor = static_cast<uint8_t>(*ptr++);
+
+ // num_exceptions: 2 bytes
+ std::memcpy(&result.num_exceptions, ptr, sizeof(result.num_exceptions));
+
+ return result;
+}
+
+// ----------------------------------------------------------------------
+// AlpEncodedForVectorInfo implementation (templated, 5/9 bytes)
+
+template <typename T>
+void AlpEncodedForVectorInfo<T>::Store(arrow::util::span<char> output_buffer) const {
+ ARROW_CHECK(output_buffer.size() >= GetStoredSize())
+ << "alp_for_vector_info_output_too_small: " << output_buffer.size() << " vs "
+ << GetStoredSize();
+
+ char* ptr = output_buffer.data();
+
+ // frame_of_reference: 4 bytes for float, 8 bytes for double
+ std::memcpy(ptr, &frame_of_reference, sizeof(frame_of_reference));
Review Comment:
safe load?
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
Review Comment:
This appears like we are hard-coding vector size. If we are going to hard-code it here, does this mean we don't have test data for multiple vector sizes? if so we should either modify the code and generate test data for different sizes or for now say say in the spec that value must be the fixed size.
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
+ header.num_elements = static_cast<uint32_t>(element_count);
+
+ std::memcpy(encoded_header, &header.compression_mode, 3);
Review Comment:
It isn't clear to me that this is always safe? Could we copy each element? does that impact performance?
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
+ header.num_elements = static_cast<uint32_t>(element_count);
+
+ std::memcpy(encoded_header, &header.compression_mode, 3);
+ std::memcpy(encoded_header + 3, &header.num_elements, sizeof(header.num_elements));
+ *comp_size = AlpHeader::kSize + compression_progress.num_compressed_bytes_produced;
+}
+
+template <typename T>
+void AlpCodec<T>::Encode(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, std::optional<AlpMode> enforce_mode) {
Review Comment:
I would maybe leave mode out now if only one values is accepted?
##########
cpp/src/arrow/util/alp/alp_wrapper.h:
##########
@@ -0,0 +1,185 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// High-level wrapper interface for ALP compression
+
+#pragma once
+
+#include <cstddef>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_sampler.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// AlpCodec
+
+/// \class AlpCodec
+/// \brief High-level interface for ALP compression
+///
+/// AlpCodec is an interface for Adaptive Lossless floating-Point Compression
+/// (ALP) (https://dl.acm.org/doi/10.1145/3626717). For encoding, it samples
+/// the data and applies decimal compression (Alp) to floating point values.
+/// This class acts as a wrapper around the vector-based interfaces of
+/// AlpSampler and Alp.
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+class AlpCodec {
+ public:
+ /// Type alias for the sampler result containing encoding presets
+ using AlpSamplerResult = typename AlpSampler<T>::AlpSamplerResult;
+
+ /// \brief Create a sampling preset from input data
+ ///
+ /// This samples the input data and generates an encoding preset that can be
+ /// reused for encoding. This is useful when you want to pre-compute the preset
+ /// outside of the benchmark loop or encode multiple batches with the same preset.
+ ///
+ /// \param[in] decomp pointer to the input data to sample
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \return the sampling result containing the encoding preset
+ static AlpSamplerResult CreateSamplingPreset(const T* decomp, size_t decomp_size);
+
+ /// \brief Encode floating point values using a pre-computed preset
+ ///
+ /// This encodes the data using a preset that was previously computed via
+ /// CreateSamplingPreset(). This avoids the sampling overhead during encoding.
+ ///
+ /// \param[in] decomp pointer to the input that is to be encoded
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \param[out] comp pointer to the memory region we will encode into.
+ /// Must be at least GetMaxCompressedSize(decomp_size) bytes.
+ /// \param[in,out] comp_size the actual size of the encoded data in bytes,
+ /// expects the size of comp as input. If this is too small,
+ /// this is set to 0 and we bail out.
+ /// \param[in] preset the pre-computed sampling result from CreateSamplingPreset()
+ static void EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset);
+
+ /// \brief Encode floating point values using ALP decimal compression
+ ///
+ /// \param[in] decomp pointer to the input that is to be encoded
+ /// \param[in] decomp_size size of decomp in bytes.
+ /// This needs to be a multiple of sizeof(T).
+ /// \param[out] comp pointer to the memory region we will encode into.
+ /// Must be at least GetMaxCompressedSize(decomp_size) bytes.
+ /// \param[in,out] comp_size the actual size of the encoded data in bytes,
+ /// expects the size of comp as input. If this is too small,
+ /// this is set to 0 and we bail out.
+ /// \param[in] enforce_mode reserved for future use.
Review Comment:
commented in another file, but lets leave out things for future use for now.
##########
cpp/src/arrow/util/alp/alp_wrapper.h:
##########
@@ -0,0 +1,185 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// High-level wrapper interface for ALP compression
+
+#pragma once
+
+#include <cstddef>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_sampler.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ----------------------------------------------------------------------
+// AlpCodec
+
+/// \class AlpCodec
+/// \brief High-level interface for ALP compression
+///
+/// AlpCodec is an interface for Adaptive Lossless floating-Point Compression
+/// (ALP) (https://dl.acm.org/doi/10.1145/3626717). For encoding, it samples
+/// the data and applies decimal compression (Alp) to floating point values.
+/// This class acts as a wrapper around the vector-based interfaces of
+/// AlpSampler and Alp.
+///
+/// \tparam T the floating point type (float or double)
+template <typename T>
+class AlpCodec {
+ public:
+ /// Type alias for the sampler result containing encoding presets
+ using AlpSamplerResult = typename AlpSampler<T>::AlpSamplerResult;
+
+ /// \brief Create a sampling preset from input data
+ ///
+ /// This samples the input data and generates an encoding preset that can be
+ /// reused for encoding. This is useful when you want to pre-compute the preset
+ /// outside of the benchmark loop or encode multiple batches with the same preset.
+ ///
+ /// \param[in] decomp pointer to the input data to sample
Review Comment:
naming bikeshed: just use `input` or `buffer` for input and `out` or `output` for `decomp`. Thoughts
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
+ header.num_elements = static_cast<uint32_t>(element_count);
+
+ std::memcpy(encoded_header, &header.compression_mode, 3);
+ std::memcpy(encoded_header + 3, &header.num_elements, sizeof(header.num_elements));
+ *comp_size = AlpHeader::kSize + compression_progress.num_compressed_bytes_produced;
+}
+
+template <typename T>
+void AlpCodec<T>::Encode(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, std::optional<AlpMode> enforce_mode) {
+ // Sample the data and encode with the preset
+ auto sampling_result = CreateSamplingPreset(decomp, decomp_size);
+ EncodeWithPreset(decomp, decomp_size, comp, comp_size, sampling_result);
+}
+
+template <typename T>
+template <typename TargetType>
+Status AlpCodec<T>::Decode(int32_t num_elements, const char* comp, size_t comp_size,
+ TargetType* decomp) {
+ ARROW_ASSIGN_OR_RAISE(const AlpHeader header, LoadHeader(comp, comp_size));
+ if (header.log_vector_size > AlpConstants::kMaxLogVectorSize) {
+ return Status::Invalid("ALP log_vector_size too large: ",
+ static_cast<int>(header.log_vector_size),
+ " > ", static_cast<int>(AlpConstants::kMaxLogVectorSize),
+ " (would overflow uint16_t element count)");
+ }
+ const uint32_t vector_size = header.GetVectorSize();
+ if (vector_size != AlpConstants::kAlpVectorSize) {
+ return Status::Invalid("Unsupported ALP vector_size: ", vector_size,
+ " (only ", AlpConstants::kAlpVectorSize, " is supported)");
+ }
+
+ const char* compression_body = comp + AlpHeader::kSize;
+ const uint64_t compression_body_size = comp_size - AlpHeader::kSize;
+
+ if (header.GetCompressionMode() != AlpMode::kAlp) {
+ return Status::Invalid("Unsupported ALP compression mode: ",
+ static_cast<int>(header.compression_mode));
+ }
+
+ ARROW_RETURN_NOT_OK(
+ DecodeAlp<TargetType>(num_elements, compression_body, compression_body_size,
+ header.GetIntegerEncoding(), vector_size,
+ header.num_elements, decomp)
+ .status());
+ return Status::OK();
+}
+
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, float* decomp);
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+template Status AlpCodec<double>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+
+template <typename T>
+int64_t AlpCodec<T>::GetMaxCompressedSize(int64_t uncompressed_size) {
+ ARROW_CHECK(uncompressed_size >= 0 && uncompressed_size % sizeof(T) == 0)
+ << "alp_decompressed_size_not_multiple_of_T";
+ const uint64_t element_count = static_cast<uint64_t>(uncompressed_size) / sizeof(T);
+ uint64_t max_alp_size = AlpHeader::kSize;
+
+ const uint64_t vectors_count =
+ static_cast<uint64_t>(std::ceil(static_cast<double>(element_count) / AlpConstants::kAlpVectorSize));
+
+ // Add offsets section (4 bytes per vector)
+ max_alp_size += vectors_count * sizeof(AlpConstants::OffsetType);
+
+ // Add per-vector metadata sizes: AlpInfo (4 bytes) + ForInfo (5/9 bytes)
+ max_alp_size +=
+ (AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize) * vectors_count;
+
+ // Worst case: everything is an exception, except two values that are chosen
+ // with large difference to make FOR encoding for placeholders impossible.
+ // Values/placeholders.
+ max_alp_size += element_count * sizeof(T);
+ // Exceptions.
+ max_alp_size += element_count * sizeof(T);
+ // Exception positions.
+ max_alp_size += element_count * sizeof(AlpConstants::PositionType);
+
+ return static_cast<int64_t>(max_alp_size);
+}
+
+template <typename T>
+auto AlpCodec<T>::EncodeAlp(const T* decomp, uint64_t element_count, char* comp,
+ size_t comp_size, const AlpEncodingParameters& combinations)
+ -> CompressionProgress {
+ // OFFSET-BASED LAYOUT
+ // [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Byte offsets to each vector (4B each)
+ // [AlpInfo₀ | ForInfo₀ | Data₀] ← Vector 0 (interleaved)
+ // [AlpInfo₁ | ForInfo₁ | Data₁] ← Vector 1
+ // ...
+ // [AlpInfoₙ₋₁ | ForInfoₙ₋₁ | Dataₙ₋₁] ← Vector n-1
+ //
+ // Benefits:
+ // - O(1) random access to any vector (no cumulative offset computation)
+ // - Better locality for single-vector access (metadata + data together)
+ // - Enables parallel decompression without coordination
+
+ // Phase 1: Compress all vectors and collect them
+ std::vector<AlpEncodedVector<T>> encoded_vectors;
+ const uint64_t num_vectors =
+ (element_count + AlpConstants::kAlpVectorSize - 1) / AlpConstants::kAlpVectorSize;
+ encoded_vectors.reserve(num_vectors);
+
+ uint64_t input_offset = 0;
+ for (uint64_t remaining_elements = element_count; remaining_elements > 0;
+ remaining_elements -= std::min(AlpConstants::kAlpVectorSize, remaining_elements)) {
+ const uint64_t elements_to_encode =
+ std::min(AlpConstants::kAlpVectorSize, remaining_elements);
+ encoded_vectors.push_back(AlpCompression<T>::CompressVector(
+ decomp + input_offset, static_cast<uint16_t>(elements_to_encode), combinations));
+ input_offset += elements_to_encode;
+ }
+
+ // Phase 2: Calculate sizes and offsets
+ const AlpIntegerEncoding integer_encoding = combinations.integer_encoding;
+ const uint64_t per_vector_metadata_size =
+ AlpEncodedVectorInfo::kStoredSize + GetIntegerEncodingMetadataSize<T>(integer_encoding);
+
+ // Offsets section comes first (after header, which is written by Encode())
+ const uint64_t offsets_section_size =
+ num_vectors * sizeof(AlpConstants::OffsetType);
+
+ // Calculate total size and per-vector offsets
+ std::vector<AlpConstants::OffsetType> vector_offsets;
+ vector_offsets.reserve(num_vectors);
+
+ // First vector starts right after the offsets section
+ uint64_t current_offset = offsets_section_size;
+ for (const auto& vec : encoded_vectors) {
+ // Store offset to this vector (relative to start of body, after header)
+ vector_offsets.push_back(static_cast<AlpConstants::OffsetType>(current_offset));
+ // Advance by metadata + data size
+ current_offset += per_vector_metadata_size + vec.GetDataStoredSize();
+ }
+ const uint64_t total_size = current_offset;
+
+ if (total_size > comp_size) {
+ return CompressionProgress{0, 0};
+ }
+
+ // Phase 3: Write offsets section
+ char* offset_ptr = comp;
+ for (const auto& offset : vector_offsets) {
+ std::memcpy(offset_ptr, &offset, sizeof(AlpConstants::OffsetType));
+ offset_ptr += sizeof(AlpConstants::OffsetType);
+ }
+
+ // Phase 4: Write interleaved vectors [AlpInfo | ForInfo | Data]
+ for (size_t i = 0; i < encoded_vectors.size(); i++) {
+ const auto& vec = encoded_vectors[i];
+ char* vector_start = comp + vector_offsets[i];
+
+ // Write AlpInfo
+ vec.alp_info.Store({vector_start, AlpEncodedVectorInfo::kStoredSize});
+ char* ptr = vector_start + AlpEncodedVectorInfo::kStoredSize;
+
+ // Write ForInfo (or other integer encoding metadata)
+ switch (integer_encoding) {
+ case AlpIntegerEncoding::kForBitPack: {
+ vec.for_info.Store({ptr, AlpEncodedForVectorInfo<T>::kStoredSize});
+ ptr += AlpEncodedForVectorInfo<T>::kStoredSize;
+ } break;
+
+ default:
+ ARROW_CHECK(false) << "unsupported_integer_encoding: "
Review Comment:
instead of crashing for now, lets just leave it out of the API for now?
##########
cpp/src/parquet/types.h:
##########
@@ -538,8 +538,9 @@ struct Encoding {
DELTA_BYTE_ARRAY = 7,
RLE_DICTIONARY = 8,
BYTE_STREAM_SPLIT = 9,
+ ALP = 10,
Review Comment:
bump
##########
cpp/src/parquet/encoder.cc:
##########
@@ -995,6 +999,90 @@ class ByteStreamSplitEncoder<FLBAType> : public ByteStreamSplitEncoderBase<FLBAT
}
};
+// ----------------------------------------------------------------------
+// ALP encoder (Adaptive Lossless floating-Point)
+
+template <typename DType>
+class AlpEncoder : public EncoderImpl, virtual public TypedEncoder<DType> {
+ public:
+ using T = typename DType::c_type;
+ using ArrowType = typename EncodingTraits<DType>::ArrowType;
+ using TypedEncoder<DType>::Put;
+
+ explicit AlpEncoder(const ColumnDescriptor* descr,
+ ::arrow::MemoryPool* pool = ::arrow::default_memory_pool())
+ : EncoderImpl(descr, Encoding::ALP, pool),
+ sink_{pool} {
+ static_assert(std::is_same<T, float>::value || std::is_same<T, double>::value,
+ "ALP only supports float and double types");
+ }
+
+ int64_t EstimatedDataEncodedSize() override { return sink_.length(); }
+
+ std::shared_ptr<Buffer> FlushValues() override {
+ if (sink_.length() == 0) {
+ // Empty buffer case
+ PARQUET_ASSIGN_OR_THROW(auto buf, sink_.Finish());
+ return buf;
+ }
+
+ // Call AlpWrapper::Encode() - it handles sampling, preset selection, and compression
+ const size_t decompSize = sink_.length();
+ size_t compSize = ::arrow::util::alp::AlpWrapper<T>::GetMaxCompressedSize(decompSize);
+
+ PARQUET_ASSIGN_OR_THROW(
+ auto compressed_buffer,
+ ::arrow::AllocateResizableBuffer(compSize, this->memory_pool()));
+
+ ::arrow::util::alp::AlpWrapper<T>::Encode(
+ reinterpret_cast<const T*>(sink_.data()),
+ decompSize,
+ reinterpret_cast<char*>(compressed_buffer->mutable_data()),
+ &compSize);
+
+ PARQUET_THROW_NOT_OK(compressed_buffer->Resize(compSize));
+ sink_.Reset();
+
+ return std::shared_ptr<Buffer>(std::move(compressed_buffer));
+ }
+
+ void Put(const T* buffer, int num_values) override {
+ if (num_values > 0) {
+ PARQUET_THROW_NOT_OK(
Review Comment:
This is encoding, not decoding I think. Reasons:
1. Reduce peak memory
2. Give better estimates for encoded size (i.e. the return value of EstimatedDataEncodedSize).
I left comment on a previous review about decoding directly which it seems this comment refers to. The default page size is currently [1MB](https://arrow.apache.org/docs/python/parquet.html#parquet-file-writing-options) on writes, this isn't trivial. One could argue this should be lowered to something more reasonable but in general, I don't thin we want to place a hard assumption on the 10s of KBs. Did you quantify how marginal? I understand the FP arithmetic is the primary bottleneck, but in most scenarios, I've seen most extra mem-copies have a reasonable high impact on perf (e.g. ~5%).
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
+ header.num_elements = static_cast<uint32_t>(element_count);
+
+ std::memcpy(encoded_header, &header.compression_mode, 3);
+ std::memcpy(encoded_header + 3, &header.num_elements, sizeof(header.num_elements));
+ *comp_size = AlpHeader::kSize + compression_progress.num_compressed_bytes_produced;
+}
+
+template <typename T>
+void AlpCodec<T>::Encode(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, std::optional<AlpMode> enforce_mode) {
+ // Sample the data and encode with the preset
+ auto sampling_result = CreateSamplingPreset(decomp, decomp_size);
+ EncodeWithPreset(decomp, decomp_size, comp, comp_size, sampling_result);
+}
+
+template <typename T>
+template <typename TargetType>
+Status AlpCodec<T>::Decode(int32_t num_elements, const char* comp, size_t comp_size,
+ TargetType* decomp) {
+ ARROW_ASSIGN_OR_RAISE(const AlpHeader header, LoadHeader(comp, comp_size));
+ if (header.log_vector_size > AlpConstants::kMaxLogVectorSize) {
+ return Status::Invalid("ALP log_vector_size too large: ",
+ static_cast<int>(header.log_vector_size),
+ " > ", static_cast<int>(AlpConstants::kMaxLogVectorSize),
+ " (would overflow uint16_t element count)");
+ }
+ const uint32_t vector_size = header.GetVectorSize();
+ if (vector_size != AlpConstants::kAlpVectorSize) {
+ return Status::Invalid("Unsupported ALP vector_size: ", vector_size,
+ " (only ", AlpConstants::kAlpVectorSize, " is supported)");
+ }
+
+ const char* compression_body = comp + AlpHeader::kSize;
+ const uint64_t compression_body_size = comp_size - AlpHeader::kSize;
+
+ if (header.GetCompressionMode() != AlpMode::kAlp) {
+ return Status::Invalid("Unsupported ALP compression mode: ",
+ static_cast<int>(header.compression_mode));
+ }
+
+ ARROW_RETURN_NOT_OK(
+ DecodeAlp<TargetType>(num_elements, compression_body, compression_body_size,
+ header.GetIntegerEncoding(), vector_size,
+ header.num_elements, decomp)
+ .status());
+ return Status::OK();
+}
+
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, float* decomp);
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+template Status AlpCodec<double>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+
+template <typename T>
+int64_t AlpCodec<T>::GetMaxCompressedSize(int64_t uncompressed_size) {
+ ARROW_CHECK(uncompressed_size >= 0 && uncompressed_size % sizeof(T) == 0)
+ << "alp_decompressed_size_not_multiple_of_T";
+ const uint64_t element_count = static_cast<uint64_t>(uncompressed_size) / sizeof(T);
+ uint64_t max_alp_size = AlpHeader::kSize;
+
+ const uint64_t vectors_count =
+ static_cast<uint64_t>(std::ceil(static_cast<double>(element_count) / AlpConstants::kAlpVectorSize));
+
+ // Add offsets section (4 bytes per vector)
+ max_alp_size += vectors_count * sizeof(AlpConstants::OffsetType);
+
+ // Add per-vector metadata sizes: AlpInfo (4 bytes) + ForInfo (5/9 bytes)
+ max_alp_size +=
+ (AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize) * vectors_count;
+
+ // Worst case: everything is an exception, except two values that are chosen
+ // with large difference to make FOR encoding for placeholders impossible.
+ // Values/placeholders.
+ max_alp_size += element_count * sizeof(T);
+ // Exceptions.
+ max_alp_size += element_count * sizeof(T);
+ // Exception positions.
+ max_alp_size += element_count * sizeof(AlpConstants::PositionType);
+
+ return static_cast<int64_t>(max_alp_size);
+}
+
+template <typename T>
+auto AlpCodec<T>::EncodeAlp(const T* decomp, uint64_t element_count, char* comp,
+ size_t comp_size, const AlpEncodingParameters& combinations)
+ -> CompressionProgress {
+ // OFFSET-BASED LAYOUT
+ // [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Byte offsets to each vector (4B each)
+ // [AlpInfo₀ | ForInfo₀ | Data₀] ← Vector 0 (interleaved)
+ // [AlpInfo₁ | ForInfo₁ | Data₁] ← Vector 1
+ // ...
+ // [AlpInfoₙ₋₁ | ForInfoₙ₋₁ | Dataₙ₋₁] ← Vector n-1
+ //
+ // Benefits:
+ // - O(1) random access to any vector (no cumulative offset computation)
+ // - Better locality for single-vector access (metadata + data together)
+ // - Enables parallel decompression without coordination
+
+ // Phase 1: Compress all vectors and collect them
+ std::vector<AlpEncodedVector<T>> encoded_vectors;
+ const uint64_t num_vectors =
+ (element_count + AlpConstants::kAlpVectorSize - 1) / AlpConstants::kAlpVectorSize;
+ encoded_vectors.reserve(num_vectors);
+
+ uint64_t input_offset = 0;
+ for (uint64_t remaining_elements = element_count; remaining_elements > 0;
+ remaining_elements -= std::min(AlpConstants::kAlpVectorSize, remaining_elements)) {
+ const uint64_t elements_to_encode =
+ std::min(AlpConstants::kAlpVectorSize, remaining_elements);
+ encoded_vectors.push_back(AlpCompression<T>::CompressVector(
+ decomp + input_offset, static_cast<uint16_t>(elements_to_encode), combinations));
+ input_offset += elements_to_encode;
+ }
+
+ // Phase 2: Calculate sizes and offsets
+ const AlpIntegerEncoding integer_encoding = combinations.integer_encoding;
+ const uint64_t per_vector_metadata_size =
+ AlpEncodedVectorInfo::kStoredSize + GetIntegerEncodingMetadataSize<T>(integer_encoding);
+
+ // Offsets section comes first (after header, which is written by Encode())
+ const uint64_t offsets_section_size =
+ num_vectors * sizeof(AlpConstants::OffsetType);
+
+ // Calculate total size and per-vector offsets
+ std::vector<AlpConstants::OffsetType> vector_offsets;
+ vector_offsets.reserve(num_vectors);
+
+ // First vector starts right after the offsets section
+ uint64_t current_offset = offsets_section_size;
+ for (const auto& vec : encoded_vectors) {
+ // Store offset to this vector (relative to start of body, after header)
+ vector_offsets.push_back(static_cast<AlpConstants::OffsetType>(current_offset));
+ // Advance by metadata + data size
+ current_offset += per_vector_metadata_size + vec.GetDataStoredSize();
+ }
+ const uint64_t total_size = current_offset;
+
+ if (total_size > comp_size) {
+ return CompressionProgress{0, 0};
+ }
+
+ // Phase 3: Write offsets section
+ char* offset_ptr = comp;
+ for (const auto& offset : vector_offsets) {
+ std::memcpy(offset_ptr, &offset, sizeof(AlpConstants::OffsetType));
+ offset_ptr += sizeof(AlpConstants::OffsetType);
+ }
+
+ // Phase 4: Write interleaved vectors [AlpInfo | ForInfo | Data]
+ for (size_t i = 0; i < encoded_vectors.size(); i++) {
+ const auto& vec = encoded_vectors[i];
+ char* vector_start = comp + vector_offsets[i];
+
+ // Write AlpInfo
+ vec.alp_info.Store({vector_start, AlpEncodedVectorInfo::kStoredSize});
+ char* ptr = vector_start + AlpEncodedVectorInfo::kStoredSize;
+
+ // Write ForInfo (or other integer encoding metadata)
+ switch (integer_encoding) {
+ case AlpIntegerEncoding::kForBitPack: {
+ vec.for_info.Store({ptr, AlpEncodedForVectorInfo<T>::kStoredSize});
+ ptr += AlpEncodedForVectorInfo<T>::kStoredSize;
+ } break;
+
+ default:
+ ARROW_CHECK(false) << "unsupported_integer_encoding: "
+ << static_cast<int>(integer_encoding);
+ break;
+ }
+
+ // Write data (packed values + exception positions + exception values)
+ const uint64_t data_size = vec.GetDataStoredSize();
+ vec.StoreDataOnly({ptr, data_size});
+ }
+
+ return CompressionProgress{static_cast<int64_t>(total_size),
+ static_cast<int64_t>(element_count)};
+}
+
+template <typename T>
+template <typename TargetType>
+auto AlpCodec<T>::DecodeAlp(size_t decomp_element_count,
+ const char* comp, size_t comp_size,
+ AlpIntegerEncoding integer_encoding,
+ uint32_t vector_size, uint32_t total_elements,
+ TargetType* decomp)
+ -> Result<DecompressionProgress> {
+ // OFFSET-BASED LAYOUT:
+ // [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Byte offsets to each vector (4B each)
+ // [AlpInfo₀ | ForInfo₀ | Data₀] ← Vector 0 (interleaved)
+ // [AlpInfo₁ | ForInfo₁ | Data₁] ← Vector 1
+ // ...
+ //
+ // Benefits:
+ // - O(1) random access to any vector (no cumulative offset computation)
+ // - Better locality for single-vector access (metadata + data together)
+ // - Enables parallel decompression without coordination
+
+ // Calculate number of vectors
+ const uint32_t num_vectors =
+ (total_elements + vector_size - 1) / vector_size;
+
+ if (num_vectors == 0) {
+ return DecompressionProgress{0, 0};
+ }
+
+ const uint64_t offsets_section_size =
+ static_cast<uint64_t>(num_vectors) * sizeof(AlpConstants::OffsetType);
+ if (comp_size < offsets_section_size) {
+ return Status::Invalid("ALP compressed buffer too small for offsets section: ",
+ comp_size, " < ", offsets_section_size);
+ }
+
+ // Read all offsets
+ std::vector<AlpConstants::OffsetType> vector_offsets(num_vectors);
Review Comment:
before the allocation we should do a sanity check on the overall buffer size relative to number of vectors to avoid OOMs on corrupted/malicious data?
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
+ header.num_elements = static_cast<uint32_t>(element_count);
+
+ std::memcpy(encoded_header, &header.compression_mode, 3);
+ std::memcpy(encoded_header + 3, &header.num_elements, sizeof(header.num_elements));
+ *comp_size = AlpHeader::kSize + compression_progress.num_compressed_bytes_produced;
+}
+
+template <typename T>
+void AlpCodec<T>::Encode(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, std::optional<AlpMode> enforce_mode) {
+ // Sample the data and encode with the preset
+ auto sampling_result = CreateSamplingPreset(decomp, decomp_size);
+ EncodeWithPreset(decomp, decomp_size, comp, comp_size, sampling_result);
+}
+
+template <typename T>
+template <typename TargetType>
+Status AlpCodec<T>::Decode(int32_t num_elements, const char* comp, size_t comp_size,
+ TargetType* decomp) {
+ ARROW_ASSIGN_OR_RAISE(const AlpHeader header, LoadHeader(comp, comp_size));
+ if (header.log_vector_size > AlpConstants::kMaxLogVectorSize) {
+ return Status::Invalid("ALP log_vector_size too large: ",
+ static_cast<int>(header.log_vector_size),
+ " > ", static_cast<int>(AlpConstants::kMaxLogVectorSize),
+ " (would overflow uint16_t element count)");
+ }
+ const uint32_t vector_size = header.GetVectorSize();
+ if (vector_size != AlpConstants::kAlpVectorSize) {
+ return Status::Invalid("Unsupported ALP vector_size: ", vector_size,
+ " (only ", AlpConstants::kAlpVectorSize, " is supported)");
+ }
+
+ const char* compression_body = comp + AlpHeader::kSize;
+ const uint64_t compression_body_size = comp_size - AlpHeader::kSize;
+
+ if (header.GetCompressionMode() != AlpMode::kAlp) {
+ return Status::Invalid("Unsupported ALP compression mode: ",
+ static_cast<int>(header.compression_mode));
+ }
+
+ ARROW_RETURN_NOT_OK(
+ DecodeAlp<TargetType>(num_elements, compression_body, compression_body_size,
+ header.GetIntegerEncoding(), vector_size,
+ header.num_elements, decomp)
+ .status());
+ return Status::OK();
+}
+
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, float* decomp);
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+template Status AlpCodec<double>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+
+template <typename T>
+int64_t AlpCodec<T>::GetMaxCompressedSize(int64_t uncompressed_size) {
+ ARROW_CHECK(uncompressed_size >= 0 && uncompressed_size % sizeof(T) == 0)
+ << "alp_decompressed_size_not_multiple_of_T";
+ const uint64_t element_count = static_cast<uint64_t>(uncompressed_size) / sizeof(T);
+ uint64_t max_alp_size = AlpHeader::kSize;
+
+ const uint64_t vectors_count =
+ static_cast<uint64_t>(std::ceil(static_cast<double>(element_count) / AlpConstants::kAlpVectorSize));
+
+ // Add offsets section (4 bytes per vector)
+ max_alp_size += vectors_count * sizeof(AlpConstants::OffsetType);
+
+ // Add per-vector metadata sizes: AlpInfo (4 bytes) + ForInfo (5/9 bytes)
+ max_alp_size +=
+ (AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize) * vectors_count;
+
+ // Worst case: everything is an exception, except two values that are chosen
+ // with large difference to make FOR encoding for placeholders impossible.
+ // Values/placeholders.
+ max_alp_size += element_count * sizeof(T);
+ // Exceptions.
+ max_alp_size += element_count * sizeof(T);
+ // Exception positions.
+ max_alp_size += element_count * sizeof(AlpConstants::PositionType);
+
+ return static_cast<int64_t>(max_alp_size);
+}
+
+template <typename T>
+auto AlpCodec<T>::EncodeAlp(const T* decomp, uint64_t element_count, char* comp,
+ size_t comp_size, const AlpEncodingParameters& combinations)
+ -> CompressionProgress {
+ // OFFSET-BASED LAYOUT
+ // [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Byte offsets to each vector (4B each)
+ // [AlpInfo₀ | ForInfo₀ | Data₀] ← Vector 0 (interleaved)
+ // [AlpInfo₁ | ForInfo₁ | Data₁] ← Vector 1
+ // ...
+ // [AlpInfoₙ₋₁ | ForInfoₙ₋₁ | Dataₙ₋₁] ← Vector n-1
+ //
+ // Benefits:
+ // - O(1) random access to any vector (no cumulative offset computation)
+ // - Better locality for single-vector access (metadata + data together)
+ // - Enables parallel decompression without coordination
+
+ // Phase 1: Compress all vectors and collect them
+ std::vector<AlpEncodedVector<T>> encoded_vectors;
+ const uint64_t num_vectors =
+ (element_count + AlpConstants::kAlpVectorSize - 1) / AlpConstants::kAlpVectorSize;
+ encoded_vectors.reserve(num_vectors);
+
+ uint64_t input_offset = 0;
+ for (uint64_t remaining_elements = element_count; remaining_elements > 0;
+ remaining_elements -= std::min(AlpConstants::kAlpVectorSize, remaining_elements)) {
+ const uint64_t elements_to_encode =
+ std::min(AlpConstants::kAlpVectorSize, remaining_elements);
+ encoded_vectors.push_back(AlpCompression<T>::CompressVector(
+ decomp + input_offset, static_cast<uint16_t>(elements_to_encode), combinations));
+ input_offset += elements_to_encode;
+ }
+
+ // Phase 2: Calculate sizes and offsets
+ const AlpIntegerEncoding integer_encoding = combinations.integer_encoding;
+ const uint64_t per_vector_metadata_size =
+ AlpEncodedVectorInfo::kStoredSize + GetIntegerEncodingMetadataSize<T>(integer_encoding);
+
+ // Offsets section comes first (after header, which is written by Encode())
+ const uint64_t offsets_section_size =
+ num_vectors * sizeof(AlpConstants::OffsetType);
+
+ // Calculate total size and per-vector offsets
+ std::vector<AlpConstants::OffsetType> vector_offsets;
+ vector_offsets.reserve(num_vectors);
+
+ // First vector starts right after the offsets section
+ uint64_t current_offset = offsets_section_size;
+ for (const auto& vec : encoded_vectors) {
+ // Store offset to this vector (relative to start of body, after header)
+ vector_offsets.push_back(static_cast<AlpConstants::OffsetType>(current_offset));
+ // Advance by metadata + data size
+ current_offset += per_vector_metadata_size + vec.GetDataStoredSize();
+ }
+ const uint64_t total_size = current_offset;
+
+ if (total_size > comp_size) {
+ return CompressionProgress{0, 0};
+ }
+
+ // Phase 3: Write offsets section
+ char* offset_ptr = comp;
+ for (const auto& offset : vector_offsets) {
+ std::memcpy(offset_ptr, &offset, sizeof(AlpConstants::OffsetType));
+ offset_ptr += sizeof(AlpConstants::OffsetType);
+ }
+
+ // Phase 4: Write interleaved vectors [AlpInfo | ForInfo | Data]
+ for (size_t i = 0; i < encoded_vectors.size(); i++) {
+ const auto& vec = encoded_vectors[i];
+ char* vector_start = comp + vector_offsets[i];
+
+ // Write AlpInfo
+ vec.alp_info.Store({vector_start, AlpEncodedVectorInfo::kStoredSize});
+ char* ptr = vector_start + AlpEncodedVectorInfo::kStoredSize;
+
+ // Write ForInfo (or other integer encoding metadata)
+ switch (integer_encoding) {
+ case AlpIntegerEncoding::kForBitPack: {
+ vec.for_info.Store({ptr, AlpEncodedForVectorInfo<T>::kStoredSize});
+ ptr += AlpEncodedForVectorInfo<T>::kStoredSize;
+ } break;
+
+ default:
+ ARROW_CHECK(false) << "unsupported_integer_encoding: "
+ << static_cast<int>(integer_encoding);
+ break;
+ }
+
+ // Write data (packed values + exception positions + exception values)
+ const uint64_t data_size = vec.GetDataStoredSize();
+ vec.StoreDataOnly({ptr, data_size});
+ }
+
+ return CompressionProgress{static_cast<int64_t>(total_size),
+ static_cast<int64_t>(element_count)};
+}
+
+template <typename T>
+template <typename TargetType>
+auto AlpCodec<T>::DecodeAlp(size_t decomp_element_count,
+ const char* comp, size_t comp_size,
+ AlpIntegerEncoding integer_encoding,
+ uint32_t vector_size, uint32_t total_elements,
+ TargetType* decomp)
+ -> Result<DecompressionProgress> {
+ // OFFSET-BASED LAYOUT:
+ // [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Byte offsets to each vector (4B each)
+ // [AlpInfo₀ | ForInfo₀ | Data₀] ← Vector 0 (interleaved)
+ // [AlpInfo₁ | ForInfo₁ | Data₁] ← Vector 1
+ // ...
+ //
+ // Benefits:
+ // - O(1) random access to any vector (no cumulative offset computation)
+ // - Better locality for single-vector access (metadata + data together)
+ // - Enables parallel decompression without coordination
+
+ // Calculate number of vectors
+ const uint32_t num_vectors =
+ (total_elements + vector_size - 1) / vector_size;
+
+ if (num_vectors == 0) {
+ return DecompressionProgress{0, 0};
+ }
+
+ const uint64_t offsets_section_size =
+ static_cast<uint64_t>(num_vectors) * sizeof(AlpConstants::OffsetType);
+ if (comp_size < offsets_section_size) {
+ return Status::Invalid("ALP compressed buffer too small for offsets section: ",
+ comp_size, " < ", offsets_section_size);
+ }
+
+ // Read all offsets
+ std::vector<AlpConstants::OffsetType> vector_offsets(num_vectors);
+ std::memcpy(vector_offsets.data(), comp,
+ num_vectors * sizeof(AlpConstants::OffsetType));
+
+ // Decode each vector using its offset for O(1) random access
+ uint64_t output_offset = 0;
+ uint64_t bytes_consumed = offsets_section_size;
+
+ for (uint32_t vector_index = 0; vector_index < num_vectors; vector_index++) {
+ // Calculate number of elements in this vector
+ const uint64_t num_full_vectors = total_elements / vector_size;
+ const uint64_t remainder = total_elements % vector_size;
+ uint16_t this_vector_elements;
+ if (vector_index < num_full_vectors) {
+ this_vector_elements = static_cast<uint16_t>(vector_size);
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ this_vector_elements = static_cast<uint16_t>(remainder);
+ } else {
+ this_vector_elements = 0;
+ }
+
+ if (output_offset + this_vector_elements > decomp_element_count) {
+ return Status::Invalid("ALP decode output buffer too small: offset=",
+ output_offset, " + elements=", this_vector_elements,
+ " > capacity=", decomp_element_count);
+ }
+
+ // Validate offset is within bounds and enough buffer remains for metadata
+ const uint64_t vector_offset = vector_offsets[vector_index];
+ constexpr uint64_t kMinVectorMetadataSize =
+ AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize;
+ if (vector_offset + kMinVectorMetadataSize > comp_size) {
+ return Status::Invalid("ALP vector offset out of bounds or insufficient buffer "
+ "for metadata: offset=", vector_offset,
+ ", metadata_size=", kMinVectorMetadataSize,
+ ", buffer_size=", comp_size);
+ }
+
+ // Jump directly to this vector using its offset
+ const char* vector_start = comp + vector_offset;
+ const uint64_t remaining = comp_size - vector_offset;
+
+ // Read AlpInfo (interleaved)
+ const AlpEncodedVectorInfo alp_info =
+ AlpEncodedVectorInfo::Load({vector_start, remaining});
+ const char* ptr = vector_start + AlpEncodedVectorInfo::kStoredSize;
+
+ // Decode based on integer encoding type
+ switch (integer_encoding) {
+ case AlpIntegerEncoding::kForBitPack: {
+ // Read ForInfo (interleaved)
+ const AlpEncodedForVectorInfo<T> for_info =
+ AlpEncodedForVectorInfo<T>::Load(
+ {ptr, remaining - AlpEncodedVectorInfo::kStoredSize});
+ ptr += AlpEncodedForVectorInfo<T>::kStoredSize;
+
+ // Validate enough buffer remains for the data section
+ const uint64_t data_remaining = comp_size - static_cast<uint64_t>(ptr - comp);
+ const uint64_t data_size =
+ for_info.GetDataStoredSize(this_vector_elements, alp_info.num_exceptions);
+ if (data_size > data_remaining) {
Review Comment:
Is there a reason to not push these guards into the alp.cc code? so the abstraction could be logically consistent?
##########
cpp/src/arrow/util/alp/alp_wrapper.cc:
##########
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/alp/alp_wrapper.h"
+
+#include <cmath>
+#include <optional>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/alp/alp.h"
+#include "arrow/util/alp/alp_constants.h"
+#include "arrow/util/alp/alp_sampler.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ubsan.h"
+
+namespace arrow {
+namespace util {
+namespace alp {
+
+// ALP serialization uses memcpy for multi-byte integers (header fields,
+// offsets, frame_of_reference) and assumes little-endian byte order on disk.
+static_assert(ARROW_LITTLE_ENDIAN,
+ "ALP serialization assumes little-endian byte order");
+
+namespace {
+
+// ----------------------------------------------------------------------
+// AlpHeader
+
+/// \brief Header structure for ALP compression blocks
+///
+/// Contains page-level metadata for ALP compression. The num_elements field
+/// stores the total element count for the page, allowing per-vector element
+/// counts to be inferred (all vectors except the last have vector_size elements).
+///
+/// Note: num_elements is uint32_t because Parquet page headers use i32 for num_values.
+/// See: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
+///
+/// Note: log_vector_size stores the base-2 logarithm of the vector size.
+/// The actual vector size is computed as: 1u << log_vector_size (i.e., 2^log_vector_size).
+/// For example, log_vector_size=10 means vector_size=1024.
+/// This allows representing any power-of-2 vector size up to 2^255 in a single byte.
+///
+/// Header format (7 bytes):
+///
+/// +---------------------------------------------------+
+/// | AlpHeader (7 bytes) |
+/// +---------------------------------------------------+
+/// | Offset | Field | Size |
+/// +---------+---------------------+-------------------+
+/// | 0 | compression_mode | 1 byte (uint8) |
+/// | 1 | integer_encoding | 1 byte (uint8) |
+/// | 2 | log_vector_size | 1 byte (uint8) |
+/// | 3 | num_elements | 4 bytes (uint32) |
+/// +---------------------------------------------------+
+///
+/// Page-level layout (offset-based interleaved for O(1) random access):
+///
+/// +-------------------------------------------------------------------+
+/// | [AlpHeader (7B)] |
+/// | [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Vector offsets |
+/// | [Vector₀][Vector₁]...[Vectorₙ₋₁] ← Interleaved data |
+/// +-------------------------------------------------------------------+
+/// where each Vector = [AlpInfo | ForInfo | Data]
+///
+/// This layout enables O(1) random access to any vector by:
+/// 1. Reading the offset for target vector (direct lookup)
+/// 2. Jumping to that offset to read metadata + data together
+struct AlpHeader {
+ /// Compression mode (currently only kAlp is supported).
+ uint8_t compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ /// Integer encoding method used (currently only kForBitPack is supported).
+ uint8_t integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ /// Log base 2 of vector size. Actual vector size = 1u << log_vector_size.
+ /// For example: 10 means 2^10 = 1024 elements per vector.
+ uint8_t log_vector_size = 0;
+ /// Total number of elements in the page (uint32_t since Parquet uses i32).
+ /// Per-vector element count is inferred: vector_size for all but the last vector.
+ uint32_t num_elements = 0;
+
+ /// Size of the serialized header in bytes.
+ static constexpr size_t kSize = 7;
+
+ /// \brief Calculate the number of vectors from total elements and vector size
+ ///
+ /// \return number of vectors (full + partial if any)
+ uint32_t GetNumVectors() const {
+ const uint32_t vector_size = GetVectorSize();
+ return (num_elements + vector_size - 1) / vector_size;
+ }
+
+ /// \brief Get the size of the offsets section
+ ///
+ /// \return size in bytes of the offsets array (num_vectors * sizeof(OffsetType))
+ uint64_t GetOffsetsSectionSize() const {
+ return static_cast<uint64_t>(GetNumVectors()) * sizeof(AlpConstants::OffsetType);
+ }
+
+ /// \brief Compute the actual vector size from log_vector_size
+ ///
+ /// \return the vector size (2^log_vector_size)
+ uint32_t GetVectorSize() const { return 1u << log_vector_size; }
+
+ /// \brief Compute log base 2 of a power-of-2 value
+ ///
+ /// \param[in] value a power-of-2 value
+ /// \return the log base 2 of value
+ static uint8_t Log2(uint32_t value) {
+ ARROW_CHECK(value > 0 && (value & (value - 1)) == 0)
+ << "value_must_be_power_of_2: " << value;
+ return static_cast<uint8_t>(__builtin_ctz(value));
+ }
+
+ /// \brief Calculate the number of elements for a given vector index
+ ///
+ /// \param[in] vector_index the 0-based index of the vector
+ /// \return the number of elements in this vector
+ uint16_t GetVectorNumElements(uint64_t vector_index) const {
+ const uint32_t vector_size = GetVectorSize();
+ const uint64_t num_full_vectors = num_elements / vector_size;
+ const uint64_t remainder = num_elements % vector_size;
+ if (vector_index < num_full_vectors) {
+ return static_cast<uint16_t>(vector_size); // Full vector
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ return static_cast<uint16_t>(remainder); // Last partial vector
+ }
+ ARROW_CHECK(false) << "alp_invalid_vector_index: " << vector_index
+ << " (num_vectors=" << GetNumVectors() << ")";
+ return 0; // Unreachable, but silences compiler warning
+ }
+
+ /// \brief Get the AlpMode enum from the stored uint8_t
+ AlpMode GetCompressionMode() const {
+ return static_cast<AlpMode>(compression_mode);
+ }
+
+ /// \brief Get the AlpIntegerEncoding enum from the stored uint8_t
+ AlpIntegerEncoding GetIntegerEncoding() const {
+ return static_cast<AlpIntegerEncoding>(integer_encoding);
+ }
+};
+
+} // namespace
+
+// ----------------------------------------------------------------------
+// AlpCodec::AlpHeader definition
+
+template <typename T>
+struct AlpCodec<T>::AlpHeader : public ::arrow::util::alp::AlpHeader {
+};
+
+// ----------------------------------------------------------------------
+// AlpCodec implementation
+
+template <typename T>
+auto AlpCodec<T>::LoadHeader(const char* comp, size_t comp_size)
+ -> Result<AlpHeader> {
+ if (comp_size < AlpHeader::kSize) {
+ return Status::Invalid("ALP compressed buffer too small for header: ", comp_size,
+ " < ", AlpHeader::kSize);
+ }
+ AlpHeader header{};
+ std::memcpy(&header.compression_mode, comp, 3);
+ std::memcpy(&header.num_elements, comp + 3, sizeof(header.num_elements));
+ return header;
+}
+
+template <typename T>
+auto AlpCodec<T>::CreateSamplingPreset(const T* decomp, size_t decomp_size)
+ -> AlpSamplerResult {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ AlpSampler<T> sampler;
+ sampler.AddSample({decomp, element_count});
+ return sampler.Finalize();
+}
+
+template <typename T>
+void AlpCodec<T>::EncodeWithPreset(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, const AlpSamplerResult& preset) {
+ ARROW_CHECK(decomp_size % sizeof(T) == 0) << "alp_encode_input_must_be_multiple_of_T";
+ const uint64_t element_count = decomp_size / sizeof(T);
+
+ // Make room to store header afterwards.
+ char* encoded_header = comp;
+ comp += AlpHeader::kSize;
+ const uint64_t remaining_compressed_size = *comp_size - AlpHeader::kSize;
+
+ const CompressionProgress compression_progress =
+ EncodeAlp(decomp, element_count, comp, remaining_compressed_size,
+ preset.alp_parameters);
+
+ AlpHeader header{};
+ header.compression_mode = static_cast<uint8_t>(AlpMode::kAlp);
+ header.integer_encoding = static_cast<uint8_t>(AlpIntegerEncoding::kForBitPack);
+ header.log_vector_size = AlpHeader::Log2(AlpConstants::kAlpVectorSize);
+ header.num_elements = static_cast<uint32_t>(element_count);
+
+ std::memcpy(encoded_header, &header.compression_mode, 3);
+ std::memcpy(encoded_header + 3, &header.num_elements, sizeof(header.num_elements));
+ *comp_size = AlpHeader::kSize + compression_progress.num_compressed_bytes_produced;
+}
+
+template <typename T>
+void AlpCodec<T>::Encode(const T* decomp, size_t decomp_size, char* comp,
+ size_t* comp_size, std::optional<AlpMode> enforce_mode) {
+ // Sample the data and encode with the preset
+ auto sampling_result = CreateSamplingPreset(decomp, decomp_size);
+ EncodeWithPreset(decomp, decomp_size, comp, comp_size, sampling_result);
+}
+
+template <typename T>
+template <typename TargetType>
+Status AlpCodec<T>::Decode(int32_t num_elements, const char* comp, size_t comp_size,
+ TargetType* decomp) {
+ ARROW_ASSIGN_OR_RAISE(const AlpHeader header, LoadHeader(comp, comp_size));
+ if (header.log_vector_size > AlpConstants::kMaxLogVectorSize) {
+ return Status::Invalid("ALP log_vector_size too large: ",
+ static_cast<int>(header.log_vector_size),
+ " > ", static_cast<int>(AlpConstants::kMaxLogVectorSize),
+ " (would overflow uint16_t element count)");
+ }
+ const uint32_t vector_size = header.GetVectorSize();
+ if (vector_size != AlpConstants::kAlpVectorSize) {
+ return Status::Invalid("Unsupported ALP vector_size: ", vector_size,
+ " (only ", AlpConstants::kAlpVectorSize, " is supported)");
+ }
+
+ const char* compression_body = comp + AlpHeader::kSize;
+ const uint64_t compression_body_size = comp_size - AlpHeader::kSize;
+
+ if (header.GetCompressionMode() != AlpMode::kAlp) {
+ return Status::Invalid("Unsupported ALP compression mode: ",
+ static_cast<int>(header.compression_mode));
+ }
+
+ ARROW_RETURN_NOT_OK(
+ DecodeAlp<TargetType>(num_elements, compression_body, compression_body_size,
+ header.GetIntegerEncoding(), vector_size,
+ header.num_elements, decomp)
+ .status());
+ return Status::OK();
+}
+
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, float* decomp);
+template Status AlpCodec<float>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+template Status AlpCodec<double>::Decode(int32_t num_elements, const char* comp,
+ size_t comp_size, double* decomp);
+
+template <typename T>
+int64_t AlpCodec<T>::GetMaxCompressedSize(int64_t uncompressed_size) {
+ ARROW_CHECK(uncompressed_size >= 0 && uncompressed_size % sizeof(T) == 0)
+ << "alp_decompressed_size_not_multiple_of_T";
+ const uint64_t element_count = static_cast<uint64_t>(uncompressed_size) / sizeof(T);
+ uint64_t max_alp_size = AlpHeader::kSize;
+
+ const uint64_t vectors_count =
+ static_cast<uint64_t>(std::ceil(static_cast<double>(element_count) / AlpConstants::kAlpVectorSize));
+
+ // Add offsets section (4 bytes per vector)
+ max_alp_size += vectors_count * sizeof(AlpConstants::OffsetType);
+
+ // Add per-vector metadata sizes: AlpInfo (4 bytes) + ForInfo (5/9 bytes)
+ max_alp_size +=
+ (AlpEncodedVectorInfo::kStoredSize + AlpEncodedForVectorInfo<T>::kStoredSize) * vectors_count;
+
+ // Worst case: everything is an exception, except two values that are chosen
+ // with large difference to make FOR encoding for placeholders impossible.
+ // Values/placeholders.
+ max_alp_size += element_count * sizeof(T);
+ // Exceptions.
+ max_alp_size += element_count * sizeof(T);
+ // Exception positions.
+ max_alp_size += element_count * sizeof(AlpConstants::PositionType);
+
+ return static_cast<int64_t>(max_alp_size);
+}
+
+template <typename T>
+auto AlpCodec<T>::EncodeAlp(const T* decomp, uint64_t element_count, char* comp,
+ size_t comp_size, const AlpEncodingParameters& combinations)
+ -> CompressionProgress {
+ // OFFSET-BASED LAYOUT
+ // [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Byte offsets to each vector (4B each)
+ // [AlpInfo₀ | ForInfo₀ | Data₀] ← Vector 0 (interleaved)
+ // [AlpInfo₁ | ForInfo₁ | Data₁] ← Vector 1
+ // ...
+ // [AlpInfoₙ₋₁ | ForInfoₙ₋₁ | Dataₙ₋₁] ← Vector n-1
+ //
+ // Benefits:
+ // - O(1) random access to any vector (no cumulative offset computation)
+ // - Better locality for single-vector access (metadata + data together)
+ // - Enables parallel decompression without coordination
+
+ // Phase 1: Compress all vectors and collect them
+ std::vector<AlpEncodedVector<T>> encoded_vectors;
+ const uint64_t num_vectors =
+ (element_count + AlpConstants::kAlpVectorSize - 1) / AlpConstants::kAlpVectorSize;
+ encoded_vectors.reserve(num_vectors);
+
+ uint64_t input_offset = 0;
+ for (uint64_t remaining_elements = element_count; remaining_elements > 0;
+ remaining_elements -= std::min(AlpConstants::kAlpVectorSize, remaining_elements)) {
+ const uint64_t elements_to_encode =
+ std::min(AlpConstants::kAlpVectorSize, remaining_elements);
+ encoded_vectors.push_back(AlpCompression<T>::CompressVector(
+ decomp + input_offset, static_cast<uint16_t>(elements_to_encode), combinations));
+ input_offset += elements_to_encode;
+ }
+
+ // Phase 2: Calculate sizes and offsets
+ const AlpIntegerEncoding integer_encoding = combinations.integer_encoding;
+ const uint64_t per_vector_metadata_size =
+ AlpEncodedVectorInfo::kStoredSize + GetIntegerEncodingMetadataSize<T>(integer_encoding);
+
+ // Offsets section comes first (after header, which is written by Encode())
+ const uint64_t offsets_section_size =
+ num_vectors * sizeof(AlpConstants::OffsetType);
+
+ // Calculate total size and per-vector offsets
+ std::vector<AlpConstants::OffsetType> vector_offsets;
+ vector_offsets.reserve(num_vectors);
+
+ // First vector starts right after the offsets section
+ uint64_t current_offset = offsets_section_size;
+ for (const auto& vec : encoded_vectors) {
+ // Store offset to this vector (relative to start of body, after header)
+ vector_offsets.push_back(static_cast<AlpConstants::OffsetType>(current_offset));
+ // Advance by metadata + data size
+ current_offset += per_vector_metadata_size + vec.GetDataStoredSize();
+ }
+ const uint64_t total_size = current_offset;
+
+ if (total_size > comp_size) {
+ return CompressionProgress{0, 0};
+ }
+
+ // Phase 3: Write offsets section
+ char* offset_ptr = comp;
+ for (const auto& offset : vector_offsets) {
+ std::memcpy(offset_ptr, &offset, sizeof(AlpConstants::OffsetType));
+ offset_ptr += sizeof(AlpConstants::OffsetType);
+ }
+
+ // Phase 4: Write interleaved vectors [AlpInfo | ForInfo | Data]
+ for (size_t i = 0; i < encoded_vectors.size(); i++) {
+ const auto& vec = encoded_vectors[i];
+ char* vector_start = comp + vector_offsets[i];
+
+ // Write AlpInfo
+ vec.alp_info.Store({vector_start, AlpEncodedVectorInfo::kStoredSize});
+ char* ptr = vector_start + AlpEncodedVectorInfo::kStoredSize;
+
+ // Write ForInfo (or other integer encoding metadata)
+ switch (integer_encoding) {
+ case AlpIntegerEncoding::kForBitPack: {
+ vec.for_info.Store({ptr, AlpEncodedForVectorInfo<T>::kStoredSize});
+ ptr += AlpEncodedForVectorInfo<T>::kStoredSize;
+ } break;
+
+ default:
+ ARROW_CHECK(false) << "unsupported_integer_encoding: "
+ << static_cast<int>(integer_encoding);
+ break;
+ }
+
+ // Write data (packed values + exception positions + exception values)
+ const uint64_t data_size = vec.GetDataStoredSize();
+ vec.StoreDataOnly({ptr, data_size});
+ }
+
+ return CompressionProgress{static_cast<int64_t>(total_size),
+ static_cast<int64_t>(element_count)};
+}
+
+template <typename T>
+template <typename TargetType>
+auto AlpCodec<T>::DecodeAlp(size_t decomp_element_count,
+ const char* comp, size_t comp_size,
+ AlpIntegerEncoding integer_encoding,
+ uint32_t vector_size, uint32_t total_elements,
+ TargetType* decomp)
+ -> Result<DecompressionProgress> {
+ // OFFSET-BASED LAYOUT:
+ // [Offset₀ | Offset₁ | ... | Offsetₙ₋₁] ← Byte offsets to each vector (4B each)
+ // [AlpInfo₀ | ForInfo₀ | Data₀] ← Vector 0 (interleaved)
+ // [AlpInfo₁ | ForInfo₁ | Data₁] ← Vector 1
+ // ...
+ //
+ // Benefits:
+ // - O(1) random access to any vector (no cumulative offset computation)
+ // - Better locality for single-vector access (metadata + data together)
+ // - Enables parallel decompression without coordination
+
+ // Calculate number of vectors
+ const uint32_t num_vectors =
+ (total_elements + vector_size - 1) / vector_size;
+
+ if (num_vectors == 0) {
+ return DecompressionProgress{0, 0};
+ }
+
+ const uint64_t offsets_section_size =
+ static_cast<uint64_t>(num_vectors) * sizeof(AlpConstants::OffsetType);
+ if (comp_size < offsets_section_size) {
+ return Status::Invalid("ALP compressed buffer too small for offsets section: ",
+ comp_size, " < ", offsets_section_size);
+ }
+
+ // Read all offsets
+ std::vector<AlpConstants::OffsetType> vector_offsets(num_vectors);
+ std::memcpy(vector_offsets.data(), comp,
+ num_vectors * sizeof(AlpConstants::OffsetType));
+
+ // Decode each vector using its offset for O(1) random access
+ uint64_t output_offset = 0;
+ uint64_t bytes_consumed = offsets_section_size;
+
+ for (uint32_t vector_index = 0; vector_index < num_vectors; vector_index++) {
+ // Calculate number of elements in this vector
+ const uint64_t num_full_vectors = total_elements / vector_size;
+ const uint64_t remainder = total_elements % vector_size;
+ uint16_t this_vector_elements;
+ if (vector_index < num_full_vectors) {
+ this_vector_elements = static_cast<uint16_t>(vector_size);
+ } else if (vector_index == num_full_vectors && remainder > 0) {
+ this_vector_elements = static_cast<uint16_t>(remainder);
+ } else {
+ this_vector_elements = 0;
Review Comment:
should this exit early? or be marked as unreachable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org